{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML with Image data  - Hyperparameter Optimization\n",
    "\n",
    "\n",
    "In the previous tutorial on using AutoGluon for image data, we saw that deep learning involves tons of design decisions (we call these *hyperparameters*) that affect the performance of the resulting model. Some common hyperparameters include choices around:\n",
    "\n",
    "- [**Network Architecture**](http://d2l.ai/chapter_convolutional-modern/index.html): how many layers should our network have, how many weights in each fully-connected layer, how many filters in each convolutional layer and what size should each one be, should residual connections be employed, what activation function, etc. \n",
    "\n",
    "- **Optimizing network parameters**: [which optimization algorithm to use](https://ruder.io/optimizing-gradient-descent/), [what learning rate to use](http://d2l.ai/chapter_optimization/lr-scheduler.html), [how to set a dynamic learning rate schedule](http://d2l.ai/chapter_optimization/lr-scheduler.html), etc.\n",
    "\n",
    "- **Regularization**: To what degree should we use the following: [dropout](http://d2l.ai/chapter_multilayer-perceptrons/dropout.html?highlight=dropout), [weight decay](http://d2l.ai/chapter_multilayer-perceptrons/weight-decay.html?highlight=weight%20decay), [batch normalization](http://d2l.ai/chapter_convolutional-modern/batch-norm.html?highlight=batch%20normalization), etc.\n",
    "\n",
    "- **Transfer Learning**: Should we initialize our network from a [pretrained model](http://d2l.ai/chapter_computer-vision/fine-tuning.html) and if so [which one](https://gluon-cv.mxnet.io/model_zoo/classification.html)?\n",
    "\n",
    "- **Data Augmentation**: Should we augment our training data with synthetically-generated examples and if so using which [augmentation technique](http://d2l.ai/chapter_computer-vision/image-augmentation.html)?\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Unlike simpler models used for tabular data for which reasonable default hyperparameters exist, the complex neural networks used for image (and text) data will be limited to suboptimal accuracy unless their hyperparameters are appropriately chosen. Because it's usually unclear what hyperparameter choices will be good for a particular dataset, \n",
    "deep learning has sometimes been referred to as [*dark magic*](https://medium.com/@alitech_2017/revealing-the-dark-magic-behind-deep-learning-eae1b451354d) or [*alchemy*](https://www.youtube.com/watch?v=x7psGHgatGM).\n",
    "\n",
    "To address this issue, AutoGluon trains many networks with different hyperparameter choices for each. Instead of having to know what specific hyperparameter-values to use, you can instead specify a search space of many hyperparameters to consider, and also specify *how* to search through this hyperparameter space for the best configuration. \n",
    "If these are properly specified, you may be able to achieve superior accuracy within less training time, but even using AutoGluon's default hyperparameter search-space/search-strategy should allow you to achieve reasonable accuracy if you let `fit()` run long enough (with many trials).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we'll work with same image classification data as before. Recall that `fit()` will automatically partition the provided data into disjoint training/validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import autogluon as ag\n",
    "from autogluon import ImageClassification as task\n",
    "import gluoncv\n",
    "\n",
    "filename = ag.download('https://autogluon.s3.amazonaws.com/datasets/shopee-iet.zip')\n",
    "ag.unzip(filename)\n",
    "\n",
    "dataset = task.Dataset('data/train')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify which Pretrained Networks to consider\n",
    "\n",
    "Instead of relying on the default options, we can specify which pretrained neural network candidates AutoGluon should consider during `fit()`. \n",
    "Given such a list, AutoGluon will try training different networks from this list to identify the best-performing candidate. \n",
    "This is an example of an `autogluon.space.Categorical` search space, in which there are a limited number of values to choose from. In our example below, the options are: [ResNet18 or MobileNetV2](https://gluon-cv.mxnet.io/model_zoo/classification.html), where AutoGluon also will search over three values (0.25, 0.5, 1.0) of the latter network's own [`multiplier`](https://gluon-cv.mxnet.io/_modules/gluoncv/model_zoo/mobilenet.html#MobileNetV2) hyperparameter (which controls this network's width)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical['resnet18_v1b', AutoGluonObject]\n"
     ]
    }
   ],
   "source": [
    "@ag.func(\n",
    "    multiplier=ag.Categorical(0.25, 0.5, 1.0),\n",
    ")\n",
    "def get_mobilenet(multiplier):\n",
    "    return gluoncv.model_zoo.MobileNetV2(multiplier=multiplier, classes=4)\n",
    "\n",
    "net = ag.space.Categorical('resnet18_v1b', get_mobilenet())\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify Optimizer Hyperparameters to consider\n",
    "\n",
    "Similarly, we can construct another search-space to identify what optimization hyperparameters work best for our task. Here we demonstrate this for the [Nesterov Accelerated SGD](https://arxiv.org/pdf/1607.01981.pdf) optimizer, specifying `autogluon.space.Real` spaces that instruct AutoGluon to search within continuous-valued ranges (optionally on a log-scale). Note that you can use analogous AutoGluon code to search the hyperparameters (here, constructor-arguments) of *any* Python object that gets passed into the training function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoGluonObject -- NAG\n"
     ]
    }
   ],
   "source": [
    "from mxnet import optimizer as optim\n",
    "\n",
    "@ag.obj(\n",
    "    learning_rate=ag.space.Real(1e-4, 1e-2, log=True),\n",
    "    momentum=ag.space.Real(0.85, 0.95),\n",
    "    wd=ag.space.Real(1e-6, 1e-2, log=True)\n",
    ")\n",
    "class NAG(optim.NAG):\n",
    "    pass\n",
    "\n",
    "optimizer = NAG()\n",
    "print(optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search Algorithms\n",
    " \n",
    "Beyond specifying the space of hyperparameter configurations to consider, you can also tell AutoGluon what strategy it should employ to actually search through this space ([`autogluon.searcher`](https://autogluon.mxnet.io/api/autogluon.searcher.html)). AutoGluon supports various [search strategies](https://autogluon.mxnet.io/api/autogluon.searcher.html) to seek out the best hyperparameter values. \n",
    "This process of finding good hyperparameters from a given search space is commonly referred to as *hyperparameter optimization* (HPO) or *hyperparameter tuning*. \n",
    "After the searcher has specified which hyperparameter configurations to try next, \n",
    "AutoGluon relies on a scheduler to actually launch & orchestrate training jobs under these hyperparameter-settings ([`autogluon.scheduler`](https://autogluon.mxnet.io/api/autogluon.scheduler.html)).\n",
    "\n",
    "<img src=\"files/images/autogluon_system.png\" width=\"500\" height=\"400\">\n",
    "\n",
    "- **Searcher**: suggests promising hyperparameter configurations for the next training jobs.\n",
    "- **Scheduler**: launches the training jobs once computation resources become available and monitors training progress.\n",
    "\n",
    "The basic first-in first-out (FIFO) scheduler simply runs training jobs in the order that hyperparameter-configurations are suggested by the searcher ([`autogluon.scheduler.FIFOScheduler`](https://autogluon.mxnet.io/api/autogluon.scheduler.html#autogluon.scheduler.FIFOScheduler)). If one training job does not require all of the available compute resources, the scheduler may launch others in parallel while sufficient compute resources remain available. A straightforward HPO approach combines this FIFO scheduler with a searcher that just randomly samples hyperparameter-configurations to try next ([`autogluon.searcher.RandomSearcher`](https://autogluon.mxnet.io/api/autogluon.searcher.html#autogluon.searcher.RandomSearcher)). While it seems naive, random search can be [suprisingly competitive in practice](https://www.argmin.net/2016/06/20/hypertuning/), and is [usually better than grid search for optimizing many hyperparameters](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf).\n",
    "\n",
    "\n",
    "### Bayesian Optimization\n",
    "\n",
    "From the previously-tried (hyperparameter-configuration, validation-performance) observations, [Bayesian Optimization](https://distill.pub/2020/bayesian-optimization/) fits a probabilistic *surrogate model* to estimate the underlying function that relates each hyperparameter configuration to the resulting validation performance of a model trained under this hyperparameter configuration ([Agnihotri et al, 2020](https://distill.pub/2020/bayesian-optimization/)).  We next use these probabilistic posterior estimates to decide which hyperparameter-configuration to try in the next training job. Here we follow an *acquisition function* that controls whether we prefer to try a configuration we are confident should perform well (because we've previously observed similar configurations with good performance), or a configuration whose performance we are highly uncertain about (because we've not previously tried any remotely similar configurations). After the new training job completes, we can use the performance produced by this new hyperparameter-configuration to update our surrogate model, and then keep iterating this process.\n",
    "\n",
    "<img src=\"files/images/bayesopt.png\" width=\"500\" height=\"400\">\n",
    "\n",
    "Below we show how to use Bayesian Optimization for the HPO search with [`autogluon.searcher.SKoptSearcher`](https://autogluon.mxnet.io/api/autogluon.searcher.html#autogluon.searcher.SKoptSearcher).  You can (optionally) specify what kind of surrogate model to use (Gaussian Process, Random Forest, etc.), as well as which acquisition function to employ (eg. [Expected Improvement, Lower Confidence Bound, etc.](https://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf)).  Here we instruct AutoGluon to perform Bayesian optimization using a Random Forest surrogate model with acquisitions based on Expected Improvement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60\n",
      "scheduler_options: Key 'delay_get_config': Imputing default value True\n",
      "\n",
      "Starting Experiments\n",
      "Num of Finished Tasks is 0\n",
      "Num of Pending Tasks is 4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheduler: FIFOScheduler(\n",
      "DistributedResourceManager{\n",
      "(Remote: Remote REMOTE_ID: 0, \n",
      "\t<Remote: 'inproc://172.31.68.247/2447/1' processes=1 threads=4, memory=64.28 GB>, Resource: NodeResourceManager(4 CPUs, 1 GPUs))\n",
      "})\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66bbd227f332483997d779d3c8f76452",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=4.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{'dataset': <autogluon.task.image_classification.dataset.NativeImageFolderDataset object at 0x7f09c8c4add0>, 'net': 'resnet18_v1b', 'optimizer': <__main__.NAG object at 0x7f09c0042110>, 'loss': SoftmaxCrossEntropyLoss(batch_axis=0, w=None), 'metric': 'accuracy', 'num_gpus': 1, 'split_ratio': 0.8, 'batch_size': 64, 'input_size': 224, 'epochs': 2, 'final_fit_epochs': 2, 'verbose': True, 'num_workers': 4, 'hybridize': True, 'final_fit': False, 'tricks': Dict{'last_gamma': False, 'use_pretrained': True, 'use_se': False, 'mixup': False, 'mixup_alpha': 0.2, 'mixup_off_epoch': 0, 'label_smoothing': False, 'no_wd': False, 'teacher_name': None, 'temperature': 20.0, 'hard_weight': 0.5, 'batch_norm': False, 'use_gn': False}, 'lr_config': Dict{'lr_mode': 'cosine', 'lr_decay': 0.1, 'lr_decay_period': 0, 'lr_decay_epoch': '40,80', 'warmup_lr': 0.0, 'warmup_epochs': 0}, 'task_id': 0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "367b190dfe674b1f9f613a65439c0300",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, max=2), HTML(value=''))), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{'dataset': <autogluon.task.image_classification.dataset.NativeImageFolderDataset object at 0x7f09c00d8ed0>, 'net': MobileNetV2(\n",
      "  (features): HybridSequential(\n",
      "    (0): Conv2D(None -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "    (2): ReLU6(\n",
      "    \n",
      "    )\n",
      "    (3): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 16, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (4): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=96, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 24, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (5): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 144, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=144, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 24, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (6): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 144, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 144, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=144, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (7): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (8): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (9): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=192, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (10): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (11): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (12): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (13): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (14): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (15): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (16): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 576, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=576, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (17): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (18): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (19): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 320, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (20): Conv2D(None -> 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "    (21): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "    (22): ReLU6(\n",
      "    \n",
      "    )\n",
      "    (23): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)\n",
      "  )\n",
      "  (output): HybridSequential(\n",
      "    (0): Conv2D(None -> 4, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "    (1): Flatten\n",
      "  )\n",
      "), 'optimizer': <__main__.NAG object at 0x7f09c012ecd0>, 'loss': SoftmaxCrossEntropyLoss(batch_axis=0, w=None), 'metric': 'accuracy', 'num_gpus': 1, 'split_ratio': 0.8, 'batch_size': 64, 'input_size': 224, 'epochs': 2, 'final_fit_epochs': 2, 'verbose': True, 'num_workers': 4, 'hybridize': True, 'final_fit': False, 'tricks': Dict{'last_gamma': False, 'use_pretrained': True, 'use_se': False, 'mixup': False, 'mixup_alpha': 0.2, 'mixup_off_epoch': 0, 'label_smoothing': False, 'no_wd': False, 'teacher_name': None, 'temperature': 20.0, 'hard_weight': 0.5, 'batch_norm': False, 'use_gn': False}, 'lr_config': Dict{'lr_mode': 'cosine', 'lr_decay': 0.1, 'lr_decay_period': 0, 'lr_decay_epoch': '40,80', 'warmup_lr': 0.0, 'warmup_epochs': 0}, 'task_id': 1}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad862d8823ee4b70ab5f701049cd17b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, max=2), HTML(value=''))), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{'dataset': <autogluon.task.image_classification.dataset.NativeImageFolderDataset object at 0x7f099c4a8790>, 'net': 'resnet18_v1b', 'optimizer': <__main__.NAG object at 0x7f099c4a3390>, 'loss': SoftmaxCrossEntropyLoss(batch_axis=0, w=None), 'metric': 'accuracy', 'num_gpus': 1, 'split_ratio': 0.8, 'batch_size': 64, 'input_size': 224, 'epochs': 2, 'final_fit_epochs': 2, 'verbose': True, 'num_workers': 4, 'hybridize': True, 'final_fit': False, 'tricks': Dict{'last_gamma': False, 'use_pretrained': True, 'use_se': False, 'mixup': False, 'mixup_alpha': 0.2, 'mixup_off_epoch': 0, 'label_smoothing': False, 'no_wd': False, 'teacher_name': None, 'temperature': 20.0, 'hard_weight': 0.5, 'batch_norm': False, 'use_gn': False}, 'lr_config': Dict{'lr_mode': 'cosine', 'lr_decay': 0.1, 'lr_decay_period': 0, 'lr_decay_epoch': '40,80', 'warmup_lr': 0.0, 'warmup_epochs': 0}, 'task_id': 2}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "22f31d86285541e0801f200e5af10973",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, max=2), HTML(value=''))), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{'dataset': <autogluon.task.image_classification.dataset.NativeImageFolderDataset object at 0x7f099ffccd10>, 'net': MobileNetV2(\n",
      "  (features): HybridSequential(\n",
      "    (0): Conv2D(None -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "    (2): ReLU6(\n",
      "    \n",
      "    )\n",
      "    (3): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 16, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (4): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=96, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 24, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (5): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 144, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=144, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 24, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (6): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 144, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 144, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=144, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (7): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (8): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (9): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=192, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (10): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (11): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (12): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (13): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (14): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (15): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (16): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 576, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=576, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (17): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (18): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (19): LinearBottleneck(\n",
      "      (out): HybridSequential(\n",
      "        (0): Conv2D(None -> 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (2): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (3): Conv2D(None -> 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "        (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (5): ReLU6(\n",
      "        \n",
      "        )\n",
      "        (6): Conv2D(None -> 320, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "      )\n",
      "    )\n",
      "    (20): Conv2D(None -> 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "    (21): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "    (22): ReLU6(\n",
      "    \n",
      "    )\n",
      "    (23): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)\n",
      "  )\n",
      "  (output): HybridSequential(\n",
      "    (0): Conv2D(None -> 4, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "    (1): Flatten\n",
      "  )\n",
      "), 'optimizer': <__main__.NAG object at 0x7f099ffcc950>, 'loss': SoftmaxCrossEntropyLoss(batch_axis=0, w=None), 'metric': 'accuracy', 'num_gpus': 1, 'split_ratio': 0.8, 'batch_size': 64, 'input_size': 224, 'epochs': 2, 'final_fit_epochs': 2, 'verbose': True, 'num_workers': 4, 'hybridize': True, 'final_fit': False, 'tricks': Dict{'last_gamma': False, 'use_pretrained': True, 'use_se': False, 'mixup': False, 'mixup_alpha': 0.2, 'mixup_off_epoch': 0, 'label_smoothing': False, 'no_wd': False, 'teacher_name': None, 'temperature': 20.0, 'hard_weight': 0.5, 'batch_norm': False, 'use_gn': False}, 'lr_config': Dict{'lr_mode': 'cosine', 'lr_decay': 0.1, 'lr_decay_period': 0, 'lr_decay_epoch': '40,80', 'warmup_lr': 0.0, 'warmup_epochs': 0}, 'task_id': 3}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d132ac8f1cb14949b1971582dda077b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, max=2), HTML(value=''))), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "{'dataset': <autogluon.task.image_classification.dataset.NativeImageFolderDataset object at 0x7f099c0bfe50>, 'net': 'resnet18_v1b', 'optimizer': <__main__.NAG object at 0x7f099c0bf1d0>, 'loss': SoftmaxCrossEntropyLoss(batch_axis=0, w=None), 'metric': 'accuracy', 'num_gpus': 1, 'split_ratio': 0.8, 'batch_size': 64, 'input_size': 224, 'epochs': 2, 'final_fit_epochs': 2, 'verbose': True, 'num_workers': 4, 'hybridize': True, 'final_fit': True, 'tricks': Dict{'last_gamma': False, 'use_pretrained': True, 'use_se': False, 'mixup': False, 'mixup_alpha': 0.2, 'mixup_off_epoch': 0, 'label_smoothing': False, 'no_wd': False, 'teacher_name': None, 'temperature': 20.0, 'hard_weight': 0.5, 'batch_norm': False, 'use_gn': False}, 'lr_config': Dict{'lr_mode': 'cosine', 'lr_decay': 0.1, 'lr_decay_period': 0, 'lr_decay_epoch': '40,80', 'warmup_lr': 0.0, 'warmup_epochs': 0}, 'task_id': 4}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0950b57591e34d708f863f1318868d6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, max=2), HTML(value=''))), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving Training Curve in checkpoint/plot_training_curves.png\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEWCAYAAACe8xtsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZxkZX3v8c+3t1m6umftbmbfupF9swXcQGVxAGEwIrKpRHMJChf3KAnkRqK5hFw0JpcrEkQxgiPo1Uw0Bi8qGlF0GsEFFJkFmAWYfemepbff/eN5qvtUdXVX1UxVr7/361Wv7nPOc049p5bzq2c5zyMzwznnnCtGxUhnwDnn3NjjwcM551zRPHg455wrmgcP55xzRfPg4ZxzrmgePJxzzhXNg8coJOlTkrZJemmk8zJRSaqU1C5pYSnTlpqkL0v6i2F4nrMl/brcz1MoSZMlmaT5I52XUpL0E0lvKzDtNkmnlztPg/HgUQKSnpO0P15AXpb0JUmpQzzWAuAjwDFmdkRpczp+xdc+/ehNvB/tkq4s9nhm1mNmKTN7oZRpiyHpyKzzMkkdieVXmdnVZnZbKZ83FzN72MxOLPfzlIukWyXdnbUuIwDFNF3xtd0l6aeSXpVIv0jSSkk7YprHJL15kOc7O/E+dcTnSb6Xjbn2M7MzzOybpTz3cvHgUToXmlkKOAV4FXBTsQeQVAUsArab2ZZD3H9CihfvVHwPXiC+H/FxX3b6sfBamdkfE+c0O65+ReK8Vo9k/sape+Pr3Qg8DjwIEC/2jwK7gKPi9s8D35B0YfZBYrBNv3evBHqSn9Hs77ekCkkq65mVmAePEjOzTcD3gOMAJE2T9EVJL0raFKukKuO2qyU9KumzknYAjwD/D5gbf518Oaa7SNJT8dfQI5KOTj9fLPV8XNJvgA5JVXHdxyT9Jv7q+aKkJknfk7RX0sOSZiSO8aCklyTtjsXmYxPbvizpDknfjfv+QtKyxPZjJf2/+GvsZUl/GddXSPqEpLWStkt6QNLMXK+ZpN9LektiuSoWyU+Jvw6/Go+xS9JqSU3Fvi/xdf+6pK9J2gtcJenV8dfjrvj+/JOk6kQeTNLiuPzVuD39Gv5c0pJi08bt50n6Y3y9/zl+Bq4u9pzisVZKuin+v1zSGkk3xddvk6TzJa1IvA8fSexbKelmSeti+vskTR/keZZLWpNYfknShyT9Lp7HfZJqhsjnn0t6Jn5OvitpXmLb5yVtlLRH0i+VqIqJr+3/iHncE9//ZIn8vHhuOyV99lBew1zMrBP4CrBIoRbhY8DLZnatmW0xs31mdi/wv4DbD+U5JLVJ+htJvwT2AU1x3WVx+zHx+7hD0hYdRo1GOXjwKDGFaqfzgSfiqnuBbqAZOBk4F/izxC6nAesIv2TOAc4DNsdfJ1dLOhL4GvBBoAH4D+Dfs76olwMXANPNrDuue1s83pHAhYSA9peEX7AVwA2J/b8HtMQ8/ArI/qV+OfBJYAawBvh0PNc64GHgP4G58Rx/EPe5AbgYODNu2wncMcjL9rX4HGlvBraZ2a+AdwPTgAXALOBaYP8gx8nnrcD98XhfJ7wvHyC8Jq8FlgN/PsT+VwA3AzMJpZu/LTatwi/YBwgXo9nAeuDUQzyfXBYBXcARwK3APcAlwAnA2cCnExfujxE+j68D5sf9irkAXwKcRXjfTyOc8wDxYvhBwuewifDd+Goiyc+B4wnv778BD6aDOHAj4XN0LjAduAY4kNj3PML36hTgTyW9oYj8D0rSZMJnb42ZtRO+S9/IkfQBoEXSokN8qquAK4F6YGuO7X9NeM1OBI4FPnGIz1N6ZuaPw3wAzwHthCLt88D/AaYQ3vSDwJRE2suBH8X/rwZeyDrWG4CNieWbgQcSyxXAJuANied+T478XJlY/ibw+cTyfwe+Pci5TAcMmBaXvwzcndh+PvCHxLk8Mchxfg+clVieQ7g4VeVI2wzsBabG5fuAv47/vwf4GXBCke/H2VnrPgX8MM9+HwUejP9XxddhcVz+KnBnIu1FwO8OIe17gP9KbBPwInB1nrxNjs8xP2v9SuCm+P9yYDdQEZcb4j4nJtI/BSyP/68HXpvYtoTwC1g5nn854UKaXn4JuCSx/E/APw6S9x9lfR6r42ehKUdaxTy8Ii4/D7x5iNejNbFuFfDBQfJwK+G7uCvr0feaZqXZQqgFOCFu25jrPaL/+/LKId67o4DuHOvbgL/Ise6yQY5zVdZnZxtweqHfi1I/Rn297xhysZk9nFwh6XjCF+VF9VdnVgAbEsmS/+cyl/AFAsDMeiVtAOYl0uQ6xsuJ//fnWE7FPFYSShJvJ1xsemOa2YQLEYQLRdq+9L6E0sDaQfK9CPiWpN7Euh5CQN2UTGhmayT9HrhQ0r8TLrYnx83/Gp9nZaxS+SrwV2bWNcjzDiXjdZJ0FKHK4ZXAVEIQ+MUQ+w/2OhSTdm4yH2ZmkjYm8vQM/e/tOWb28yGeI5etZpZ+zdMltAHvvcIHcgHwH5KSo6NWEEoA2wp4ruxznD1IukXAnZKSJc9uQmnnZUk3En5IHUG4EE8GZkv6I+G1GOwzlisPQ70n/2pmfaX+WLrILsVmpEnYRvgBlG1OYvuhGPT7r9CQ/4/AawjnlX3tGFFebVVeGwi/ZGab2fT4qDezYxNp8g1rvJnw5QMg8aVPXoAPZ2jkK4AVhCqNacDi9FMVsO8GYNkQ285LnPd0M5tsoU0ol3TV1QrgaTNbA2BmXWb2STM7hvAlegvwrgLylkv26/QF4HdAs5nVE6oIyt1o+SLhogn0vZ99PwTMLNkgXmzgKJiFn66bgDfleI8O9UI4mA2EX+3J55liZo9LOodQEn4r4Vf8TMIFXYk8DvYZG04PE6rpsl0KrDWz53NsK8RQ393bCT/gjomfz2sp/+ezYB48ysjMXgS+D9wuqV6hEXmZpDOLOMwDwAWSzor1wB8hBKSflSibdfF42wm/vv+uiH2/Axwh6YOSJkmqk3Ra3HYnoX59EYCkBkkrhjjWSkK99vsI7RLE/d4o6fhYQtpDqO7oKSKPQ6kjfDk7FDohDNXeUSrfAU6RdKFCj68PEEp8I+FO4NbYToekRuXoOVSi57lJ0ivi88xQ/70MdYT3dCtQA9xCKHmk3Q38naSlCk4erFG/zP4BmCPpzvg6TZH0LkK70UfL9Jx1hOrcPQqdMT5Upuc5JB48yu9dhC/F04RG42+Qu/ibk5k9Q6jr/GdC0fhCQjfUzhLl7yuEarFNMY+PFZG3vYSGxAsJ1QfPAm+Mmz9HqIP+vkLvpscIjaqDHetFQsPpawiN2WlHEF6zPYR2lB+T2dh6OD5CaBTdSyiFfH3o5IfPzF4G3gF8hhCwlxEakA+W+7lzuI3wi/qH8T36GaHhuaTM7GvA/wb+r6Q9wJOEzw3AvwM/IVRNrSN8xpMNx7cC3wV+SPgM3AlMKnUe84nv22sJJaNnCPm8DrjUzL5dpqe9ifB92kP4DjxYpuc5JIoNL865ERBLVJsJjc//NdL5ca5QXvJwbpjFeyamSZpE6E3XDfxyhLPlXFE8eDg3/F5HfxXNckJPvZGotnLukHm1lXPOuaJ5ycM551zRJsxNgrNnz7bFixePdDacc25Mefzxx7eZ2YDu5BMmeCxevJi2traRzoZzzo0pknLeAOnVVs4554rmwcM551zRPHg455wrmgcP55xzRfPg4ZxzrmgePJxzzhXNg4dzzrmiTZj7PJxzbqLYva+LNVv3smZLO+u2dfCJ5UeRmM20JDx4OOfcGGRmbGvvZM2WdtZsCYHi2fjYurd/nM1JVRW893VLaKybPMTRiufBwznnRjEz48XdB3h2S3tfoHj25XbWbG1n176uvnSpSVU0N6Y488gGWhpTNDemaGmsY96MKVRWlH72Wg8ezjk3CvT0Ght27EuUIPayNgaMjs7+mZdnTK2mpbGO84+fQ3NDipamECSa6ieVvGpqKB48nHNuGHV29/L89o4QIGIJ4tmX97JuWwed3b196ZrqJ9HSWMfbWxfQ3FeSSDErNeyz8ObkwcM558pgf2cPa7emq5pCSWLNlnae276Pnt4wj5IE82dMobkhxRlHNvQFiebGFPWTq0f4DIbmwcM55w7DngNdfQEiGSg27txPeq69ygqxaNZUWhpTnHfcnL4AsawhxZSaypE9gUPkwcM55wqwvf1gX3tEMlC8tOdAX5qaqgqWzq7lpAUzuOSUBbQ0hSCxeFYtNVXj67Y6Dx7OOReZGS/vOdhXxfTslnbWxHaJHR2dfemm1lTS0pjiNc2zaGms62uPWDBzall6No1GHjyccxNOb6+xced+1mwN3V7TpYm1W9rZe7C7L920KdW0NKZ487FNLGtI0dJUR0tjijnTJg9rz6bRqOzBQ9Jy4HNAJXC3md06SLpLgAeBV5lZm6RzgFuBGqAT+JiZ/TCmfQSYA+yPu59rZlvKeiLOuTGnq6eX57fvy7g34tmX21m3rZ0DXf09mxrqJtHSmOKtp8yL90iE0sTsVM2EDxKDKWvwkFQJ3AGcA2wEVktaZWZPZ6WrA24AfpFYvQ240Mw2SzoOeAiYl9h+pZn5vLLOOQ509bBua0ffvRHpksT6bR10x55NAPOmT6G5McVrls3qa49obqhj2tTR3bNpNCp3yeNUYI2ZrQOQtBJYATydle5vgduAj6ZXmNkTie1PAZMlTTKzgzjnJqT2g90ZPZrSgeKFHfv6ejZVCBbNqqW5McU5xzT13Wm9tKGW2kleU18q5X4l5wEbEssbgdOSCSSdDCwws+9I+ii5vQ14IitwfElSD/BN4FNmZrl3dc6NNTs7OvuqmJL3SLy4O9GzqbKCJbNrOW7eNC4+aV5fSWLJ7FomVY3N7q9jSbmDR67Kwr6LvKQK4LPA1YMeQDoW+Hvg3MTqK81sU6zu+ibwTuArOfa9BrgGYOHChYeQfedcuZgZW/ce7Ktieja2S6zd2s629v6eTVOqK2luTHH60lkZd1ovnDmVqsrx1f11LCl38NgILEgszwc2J5brgOOAR2Kj1BHAKkkXxUbz+cC3gHeZ2dr0Tma2Kf7dK+l+QvXYgOBhZncBdwG0trZ6ycS5EdDba2zatZ81W0O312Q32L0H+ns21U2uoqUxxVlHhaqm5qYUzQ0p5k2fQsUE6f46lpQ7eKwGWiQtATYBlwFXpDea2W5gdno59qL6aAwc04HvAjea2aOJNFXAdDPbJqkaeAvwcJnPwzmXR3dPL8/Hgf0y2yU62N/VP7Df7FQNzY0pVpw0l5bGur4RYBvqhndgP3d4yho8zKxb0vWEnlKVwD1m9pSkW4A2M1s1xO7XA83AzZJujuvOBTqAh2LgqCQEjn8p20k45zIc7O5h/baOvvaINYmeTZ09/d1f506bzLLGFJefmuzZlGJGbc0I5t6ViiZKO3Nra6u1tXnPXucK1XGwu29gv+SQHM9v7yDd+1WChTOnZtwb0dKYYlljipT3bBoXJD1uZq3Z6/3ddW6CS09Z2t+zKfzdtGt/X5rqSrF4Vi1Hz6njwhPm0NxUR3NDiqUNtUyu9p5NE5EHD+cmgPSUpcl7I9J3XGdPWdrcmKJ18Qwub0zPI1HHollTqfaeTS7Bg4dz44iZsXn3gVCCeHlvouG6nd37+6csrZtUxbLGFG+Ic0ikZ6Pznk2uUB48nBuDenqNF/qmLO0PEmuzpiydWRt6Nl1wwhxa4p3WzY2pYZ+y1I0/HjycG8U6u3t5bnvHgDuts6csPaJ+Ms2NKd7eGueQaAi9m0bLlKVu/PHg4dwokJyyNHkT3fNZU5YumDGV5sYUZx7ZwLJEz6bRPmWpG388eDg3jPqmLO0bHnwvz8aeTckpSxfPmsqRjXWcf9wcWprCdKVjecpSN/548HCuDLa3H8y4NyJdmnh5T3/PppqqCpY1pDh54QwubV3Qd4/EonE4Zakbfzx4OHeIzIyX9hzIuD9ibQwUO/f192yqrQkD+72uuaEvQLQ0pZg/Y+JMWerGHw8ezuXR02ts2rk/jPq6JTNQtCemLJ0+NUxZuvy4I2hOjNnkU5a68ciDh3NRmLK0Y8Cd1mu3tnMw0bOpsW4SzY0p3nbKvL47rVuaUsyq9SlL3cThwcNNOAe6+ns2JW+iey5rytL5M8KUpa9tntV3p3VzY4ppU7xnk3MePNy4tfdAF2u3doQ7rfvmkmhnw859GT2bFs0M3V/PPaYp3iNRx7LGWqbW+NfDucH4t8ONeTs7OsNYTYk7rXNNWbq0oZbj50/jT06Z1zev9eLZU33KUucOgQcPNyaYGVv2HoztEXszusFu7+ifsnRqTSXLGlK8eumsvpvoWprqWDBjik9Z6lwJefBwo0rflKVZd1qvyZqytH5yFS1NdZx9dFP/REONKeZO84H9nBsOZQ8ekpYDnyPM+ne3md06SLpLgAeBV5lZW1x3I/BeoAe4wcweKuaYbvRKT1n67MuhN1O6XWLglKWTaGlMcfFJ8/rHbGpK0ZDygf2cG0llDR6SKoE7gHOAjcBqSavM7OmsdHXADcAvEuuOIcx5fiwwF3hY0pFxc95jutHhQFeYsjT7Jrr12zro6unv2TRv+hSWNaY47bRZfTfSNTemmD7Vpyx1bjQqd8njVGCNma0DkLQSWAFkX+j/FrgN+Ghi3QpgpZkdBNZLWhOPR4HHdMMoPWXps31jNoW2iRd27OubsrQiTlna3FjHm45q6gsQPmWpc2NPub+x84ANieWNwGnJBJJOBhaY2XckfTRr38ey9p0X/x/ymIljXwNcA7Bw4cJDyb/LsmtfZ0Y7RLo0kT1l6ZLZtRwzt56LTprXV5JYMtunLHVuvCh38MhVKd1XVyGpAvgscHUR++bqMmM51mFmdwF3AbS2tuZM4wYyM7a2H0yM/NofKLa19w/sN7k6DOz3qr4pS+toaUqxcKZPWerceFfu4LERWJBYng9sTizXAccBj8TGzyOAVZIuyrPvUMd0BertNTbv3j/gTus1OaYsbW5K8cZXNPRNV9rcmPIpS52bwModPFYDLZKWAJsIDeBXpDea2W5gdnpZ0iPAR82sTdJ+4H5JnyE0mLcAvySUSAY9phuou6eXDTv3Z9xpvSYOz7EvMWXprDhl6VvilKXpkkRjnfdscs5lKmvwMLNuSdcDDxG61d5jZk9JugVoM7NVQ+z7lKQHCA3h3cB1ZtYDkOuY5TyPseJgdw/Pbeuf1zrdHrFuawedPf0D+82ZFqYsfcerFvTdad3cmGJmrfdscs4VRmYToymgtbXV2traRjobJbGvs5t1WztCgHi5v8rp+R0DpyxN92hKPup8ylLnXIEkPW5mrdnrvX/kKLZ7f5yyNOtO6407+3s2VVWIxbNrObKpjgtOmNMXIJY1pLxnk3OubDx4jDAzY3tHZ9/9EWtiu8SzL7ezZW9/z6ZJVRUsbUhxSpyyND0b3cKZPmWpc274efAYJmbGi7sPZJQg0iWKAVOWNtVxxpENGXda+5SlzrnRxINHifX0Ght37su803pr7ilLj2ysY/lxc/oCREtTiiPqfcpS59zo58HjEHV2hylLs++2Xpc1ZWlTfZiy9JJXzu8bIry50acsdc6NbR488khOWdp/p/Vent++b8CUpS2NKV7XPIuWxjqWxSDhU5Y658ajvMFD0ilDbTezX5UuO6PPhf/8U57d0g7EKUtnTaW5IcXy447ouz9iaYNPWeqcm1gKueLdHv9OBlqBXxPu8j6BMIT668qTtdHhhrNaqJBoaUqxaJZPWeqcc1BA8DCzN0Lf0OfXmNlv4/JxZA6hPi5deOLckc6Cc86NOsXcIHBUOnAAmNnvgJNKnyXnnHOjXTEV9X+QdDfwVcIQ6FcBvy9Lrpxzzo1qxQSPq4H3AR+Iyz8BPl/qDDnnnBv9CgoecS7yu83sKsLkTc455yawgto84lDoDZJ8zG7nnHNFVVs9BzwqaRXQkV5pZp8pdaacc86NbsUEj83xUUGYPtY559wEVXDwMLNPHsoTSFoOfI4w69/dZnZr1vZrgeuAHqCdcC/J05KuBD6WSHoCcIqZPRmnq50DpCe2ONfMthxK/pxzzhWv4OAhqQH4C+BYwt3mAJjZm4bYpxK4AzgH2AislrTKzJ5OJLvfzO6M6S8CPgMsN7P7gPvi+uOBfzOzJxP7XWlm42NqQOecG2OKuUnwPuAPwBLgk4Q2kNV59jkVWGNm68ysE1gJrEgmMLM9icVawj0k2S4HvlZEXp1zzpVRMcFjlpl9Eegysx+b2XuA0/PsMw/YkFjeGNdlkHSdpLXAbcANOY7zDgYGjy9JelLSzRpkbHNJ10hqk9S2devWPFl1zjlXqGKCR3q6uxclXSDpZGB+nn1yXdQHlCzM7A4zWwZ8HLgp4wDSacC+OBxK2pVmdjzw+vh4Z64nN7O7zKzVzFobGhryZNU551yhigken5I0DfgIYUDEu4EP5dlnI7AgsTyf0GNrMCuBi7PWXUZWqcPMNsW/e4H7CdVjzjnnhkkxXXUfNrMDwG7gjQXusxpokbQE2EQIBFckE0hqMbNn4+IFwLOJbRXA24EzEuuqgOlmtk1SNfAW4OEizsM559xhKiZ4/E7Sy8B/Eca1etTMdg+1g5l1S7oeeIjQVfceM3tK0i1Am5mtAq6XdDahWmwn8O7EIc4ANprZusS6ScBDMXBUEgLHvxRxHs455w6TzHJ1bhoksbSQ0MbwWuB8YJeZjYlh2VtbW62tzXv2OudcMSQ9bmat2euLuc9jPiFovB44EXgK+GnJcuicc27MKKba6gVCG8bfmdm1ZcqPc865MaCY3lYnA18BrpD0c0lfkfTeMuXLOefcKFbM2Fa/jjfyrSVUXV1FaND+Ypny5pxzbpQqps2jjdDT6WeEto4zzOz5cmXMOefc6FVMm8d5ZuZjfDjnnCuqzaNC0hclfQ9A0jHe5uGccxNTMcHjy4Sb/ebG5T8CHyx1hpxzzo1+xQSP2Wb2ANAL4e5xwgROzjnnJphigkeHpFnEUXElnU4Y58o559wEU0yD+YeBVcAySY8CDcAlZcmVc865Ua2g4BFHt50MnAm8gjBPxzNm1jXkjs4558algoKHmfVKut3MXk0Y08o559wEVkybx/clvW2wKV+dc85NHMW2edQC3ZIOEKquzMzqy5Iz55xzo1YxY1vVDbVd0rFm5lVazjk3ARRTbZXPv+ZaKWm5pGckrZH0iRzbr5X0W0lPSvqppGPi+sWS9sf1T0q6M7HPK+M+ayT9k1elOefc8Cpl8BhwAZdUCdwBnAccA1yeDg4J95vZ8XFGwtuAzyS2rTWzk+IjOYfI54FrgJb4WF7C83DOOZdHKYNHrvlsTwXWmNk6M+sEVgIrMnYy25NYrB3kOH0kzQHqzeznFubQ/Qpw8WHl3DnnXFFKGTxymQdsSCxvjOsySLouzhVyG3BDYtMSSU9I+rGk1yeOuTHfMeNxr5HUJqlt61YfENg550qllMGjM8e6XG0RA0oWZnaHmS0DPg7cFFe/CCw0s5MJPb3ul1Rf6DHjce8ys1Yza21oaCjkHJxzzhWgmK66SJoHLEruZ2Y/iX9Pz7HLRmBBYnk+sHmIp1hJaM/AzA4CB+P/j8eSyZHxmPOLOKZzzrkSK2Ymwb8H3gE8Tf9ougb8ZIjdVgMtkpYAm4DLgCuyjttiZs/GxQuAZ+P6BmCHmfVIWkpoGF9nZjsk7Y0DM/4CeBfwz4Weh3POucNXTMnjYuAVsURQEDPrlnQ9YR6QSuAeM3tK0i1Am5mtAq6XdDbQBewE3h13PwO4RVJ66PdrzWxH3PY+wvwiU4DvxYdzzrlhotBhqYCEYQbBt5tZe3mzVB6tra3W1tY20tlwzrkxRdLjZtaavb6Yksc+4ElJPyC2RQCY2Q2D7+Kcc248KiZ4rIoP55xzE1wxY1vdK6mG0OMJfD4P55ybsIrpbfUG4F7gOcK9FgskvTvdVdc559zEUUy11e3AuWb2DICkI4GvAa8sR8acc86NXsXcYV6dDhwAZvZHoLr0WXLOOTfaFVPyaJP0RfqHXr8SeLz0WXLOOTfaFRM83gdcRxi4UIQ7y/9POTLlnHNudCumt9VBwlwbn8mX1jnn3PiWN3hIesDMLpX0W3KPiHtCWXLmnHNu1Cqk5PGB+Pct5cyIc865sSNvbyszezH++34zez75AN5f3uw555wbjYrpqntOjnXnlSojzjnnxo5C2jzeRyhhLJX0m8SmOuDRcmXMOefc6FVIm8f9hPky/ifwicT6vYn5NZxzzk0ghbR57Daz58zs8tjOsZ/Q6yolaWG+/SUtl/SMpDWSPpFj+7WSfivpSUk/lXRMXH+OpMfjtsclvSmxzyPxmE/GR2NRZ+2cc+6wFDMw4oWEezzmAlsIc5n/Hjh2iH0qgTsI7SUbgdWSVpnZ04lk95vZnTH9RfE5lgPbgAvNbLOk4wizEc5L7HelmfnsTs45NwKKaTD/FHA68EczWwKcRf42j1OBNWa2zsw6gZXAimQCM9uTWKwl3ktiZk+Y2ea4/ilgsqRJReTXOedcmRQTPLrMbDtQIanCzH4EnJRnn3nAhsTyRjJLDwBIuk7SWuA2wvAn2d4GPJE1f/qXYpXVzZKU68klXSOpTVLb1q1b82TVOedcoYoJHrskpQhjWt0n6XNAd559cl3Uc92lfoeZLQM+DtyUcQDpWODvgT9PrL7SzI4HXh8f78z15GZ2l5m1mllrQ0NDnqw655wrVDHBYwVhHvMPAf8JrAUuzLPPRmBBYnk+sHmQtBCqtS5OL0iaD3wLeJeZrU2vN7NN8e9eQm+wUws+C+ecc4etmODRCNSYWbeZ3Qv8C+Fej6GsBlokLYlT2F5G1jzokloSixcAz8b104HvAjea2aOJ9FWSZsf/qwnDpvyuiPNwzjl3mIoJHg8CvYnlnrhuUGbWDVxP6Cn1e+ABM3tK0i2xZxXA9ZKekvQk8GHg3en1QDNwc1aX3EnAQ/GGxSeBTYRA5pxzbpjIbEATRO6E0pNmdlLWul+b2YllyVmJtba2Wlub9+x1zrliSHrczFqz1xdT8tiaKC0gaQXhXgznnHMTTDEzCV5L6GX1vwm9qDYA7ypLrpxzzo1qxcwkuBY4PXbXVQHnPewAABc/SURBVOzp5JxzbgIqZFTdq8zsq5I+nLUeADPzaWmdc26CKaTkMTX+zdct1znn3ARRSPBYFv8+bWZDds11zjk3MRTS2+r8eDPejeXOjHPOubGhkJLHfxK65NZKSo6AK8DMrL4sOXPOOTdqFTIZ1MfMbBrwXTOrTzzqPHA459zEVPBNgma2In8q55xzE0He4CHpp/HvXkl74t/0Y0++/Z1zzo0/eds8zOx18a931XXOOQcUUW0laVl6GlhJb5B0Qxw23Tnn3ARTzMCI3wR6JDUDXwSWECZics45N8EUEzx64/wcbwX+0cw+BMwpT7acc86NZsUEjy5JlxMma/pOXFdd+iw555wb7YoJHn8KvBr4tJmtl7QE+Gq+nSQtl/SMpDWSPpFj+7WSfhtnCvyppGMS226M+z0j6c2FHtM551x5FTyTYMZO0gxggZn9Jk+6SuCPwDnARsKc5peb2dOJNPVmtif+fxHwfjNbHoPI14BTgbnAw8CRcbchj5mLzyTonHPFO+yZBCU9Iqle0kzg18CXJOUbjv1UYI2ZrTOzTmAlkHGzYTpwRLVAOpqtAFaa2UEzWw+sicfLe0znnHPlVUy11bR4of8T4Etm9krg7Dz7zCPMOJi2Ma7LIOk6SWuB24Ab8uxb0DHjca+R1CapbevWrXmy6pxzrlDFBI8qSXOAS+lvMM9HOdYNqCczszvMbBnwceCmPPsWdMx43LvMrNXMWhsaGgrMsnPOuXyKCR63AA8RqoxWS1oKPJtnn43AgsTyfGDzEOlXAhfn2bfYYzrnnCuxYgZGfNDMTjCz98fldWb2tjy7rQZaJC2RVANcBqxKJpDUkli8gP6AtAq4TNKk2LOrBfhlIcd0zjlXXoXM5wGApMnAe4Fjgcnp9Wb2nsH2MbNuSdcTSiyVwD1m9pSkW4A2M1sFXC/pbKAL2Em4j4SY7gHgaaAbuM7MemJeBhyziHN2zjl3mAruqivpQeAPwBWEKqwrgd+b2QfKl73S8a66zjlXvMPuqgs0m9nNQIeZ3UuoYjq+VBl0zjk3dhQ1PEn8u0vSccA0YHHJc+Scc27UK7jNA7gr3ll+M6GBOgX8dVly5ZxzblQrOHiY2d3x3x8DS8uTHeecc2NB3uAh6cNDbTezfEOUOOecG2cKKXmkp5/NdXd38aMqOuecK5vOnk72du5lb+de2rva2dO5h9PnnE6Fimnizq+QOcw/CSDpXuADZrYrLs8Abi9pbpxzbgIzM/Z17+u7+CcDQHtne//6rrgtrtvTuYf2rvD/wZ6DA4776OWPUl9TX9K8FtNgfkI6cACY2U5JJ5c0N845N4Z19XbR3tlOe2c7e7qyLvjxot/emQgGXXsHBIpe6x3yOaorqqmrqaO+pp66mjpS1SmOqD2Cupq6vkeqOpWxPKVySsnPtZjgUSFphpntBIhDsxezv3POjVpmxv7u/X2/4JMX9L5f9+lgkLjoJ9ft796f93lqq2v7L+zVdTRNbWLZ9GXUVddlBoCaFPXV9X3/p9dPqpw0DK9GfsVc/G8HfibpG4S2jkuBT5clV845V6Se3p6+C312AMhV1ZNrXbd1D/kcVarKuLjX1dTRMKUh84JfUz/gl3+6NJCqTlFZUTlMr0h5FdNV9yuS2oA3ERrO/yTf7H3OOVeogz0HM3/xx6qfoS74ybr+jq6OvM8xpWpK3y/+upo6Zk2exaL6RUNe8Otr6vsCxeTKyUi5ZoWYeIqqdorBwgOGcy5Dr/XS0dUx4IKer/on2Rjc2ds55HNUqCLjgl5XU8fCuoU56/qTF/x0oKitqaW6onqYXpFRorcX2l+C+rklP7S3WTjn6OrpGvCLvuDqn9gIbHl67k+unJxRdz9t0jTm183vW5fr138yGEypmuK/+nPp7YU9m2DHOtixNv5dD9vXws710H0AbtwEk1IlfVoPHs6NcemG3qF68PQFg6wG3/TygZ4DQz6HUMaFPVWTYk5qDkdWH5mzqifXr//qygn2q7+UentCgNieDg7Jx3pIds+tnAQzl8DMZdB8Fswsz4AgHjycG2Hdvd0DLvqFVP8kf/X3hKluBpXs3pm+uDdNbcro7jlUfX9tdW3JbzJzWXp7YPeG/qCwPREgdq6HnkS1XtXkEBRmNUPLOSFQzFwaHvXzoKL875UHD+cOg5lxoOdAzpu1Cqnr39tZXPfO9AW9cWojS6cvzeje2Vf9U12fUT00mrp3Tng93ZkBYse6/tLEzuegt6s/bdWUEAxmt8ArlsfgEINE3ZxhCRBDKXvwkLQc+Bxh1r+7zezWrO0fBv6MMFvgVuA9Zva8pDcCn00kPQq4zMy+LenLwJnA7rjtajN7srxn4sajnt4eOro7hv51n+dO3+7ewrp3Ji/os6fMzlyX1cc/WRoYT907J4Sebtj9QqLkkKhq2vl8ZoConhqCQePRcNQFMCtRgqibA6O4jaeswUNSJXAHcA6wEVgtaVVWF98ngFYz2yfpfcBtwDvM7EfASfE4M4E1wPcT+33MzL5Rzvy70a+zp3PI+vycDb+J7e1d7XmfY0rVlIyL+4zJM1hYv7BvXbpv/2DVP969cxzq6YJdL2SWHNKBYtcLkPxBUV0Ls5ZC07Fw9EX9wWHWMkg1jeoAMZRylzxOBdaY2ToASSuBFSS6+8YgkfYYcFWO41wCfM/M9pUxr26Y9Vov+7r2Ddl3P19poJDundkNuAtSC4a84Cfv7J2Q3Ttd0N0ZA8TagVVMu16AZDtTTV1opJ5zIhz71swqplTjmA0QQyl38JgHbEgsbwROGyL9e4Hv5Vh/GZA99PunJf018APgE2Y2YDQwSdcA1wAsXLiwiGy7QqTH8Rns5q181T8dXR15x/GZVDkpYyiH+pp65qbmZqwb6tf/1Kqp/qvfDa67M7Q17MhRxbTrBUh+PifVh2Aw92Q47m2JKqZlUDt7XAaIoZQ7eOR6NXN2Bpd0FdBKaMtIrp9DmCv9ocTqG4GXgBrgLuDjwC0Dnsjsrrid1tZWHz4+Id29c9D6/OyeP1l3+rZ3tedt6E1270zX7c+pnVPwUA51NXXUVNYM0yvixq2uA7Dr+awqpvh398asADEtVDHNeyUcf2l/9dLMpTB11oQLEEMpd/DYCCxILM8HNmcnknQ28FfAmTlKEJcC3zKzvlYmM3sx/ntQ0peAj5Y012NAd283HV0dQ1/w81T/FNq9M/kLP929MxkUBqv+8e6dbth0HYgliLVZQWJ96N2U/M06eVooLSw4DU68PLOb69SZHiAKVO7gsRpokbQE2ESofroimSAO6/4FYLmZbclxjMsJJY3kPnPM7EWF+oiLgd+VI/PlYmb94/gMMS7/UPX9+7rzN//UVtdm1Pc3TG1g6fSlA8brGaz6x7t3ulGla38IBtnVS9vXhRvokgFiyowQFBaeDrOu7A8O6QDhDltZg4eZdUu6nlDlVAncY2ZPSboFaDOzVcA/ACngwVg3/YKZXQQgaTGh5PLjrEPfJ6mBUC32JHBtOc8jW6/10t7VnnOs/oxhmnPc6Zv+vyvZXS+HKlUNuLgvrl885AU/4w5g797pxqLOfeGGuFx3Uu/ZlJl26qwQDBa/LtGDaSnMWOIBYhjIbGI0BbS2tlpbW1vR+336sU/zx51/7B/qOXbvzDeOT7J7Z66+/Pnq+30cHzdudXbkuElufShN7H0xM+3U2Zn3PiQfU6aPTP4nGEmPm1lr9nq/wzyPfd37qKyoZF5qXuZ4PYNM3JKqSZGqSXn3TjexHdw7sIopfdNc+0uZaWsbQzBY+sZQcujr5roktE+4UcmDRx6ffp3Pd+VcTgf2DF7F1P5yZtpUUxyo7+wQFNKliRlLYHJp59Z2w8ODh3NucAd25x6ob8da6NiambZuTggI2QP1zVxa8uHA3cjz4OHcRLd/18CSQ7o0sW9bZtq6uSEYvOK8RHCIVUw1tSOTfzciPHg4NxHs3zmw5JAOEvt3ZKatnxeCwlEXZN4kN2MJ1Ewdmfy7UceDh3Pjxb4dA0sO6SCxf2cioWDa/FBaOOai/iqmWctgxmKonjJSZ+DGEA8ezo0VZjFA5Biob8c6OLArkVgwbUHovZQ9UN+MxVA9eaTOwo0THjycG03MoGNb7oH6tq+Dg7v706oiBIiZS7MG6osBospHCHDl48HDueFmFnoq5ZoLYsd6OLinP60qYPrCEBBOuDRzLojpCz1AuBHjwcO5cjCD9i2DVDGth869/WlVGQLBrDhYX7Kb6/SFUOUjC7vRx4OHc4fKDPa+lKMHU1zu6uhPW1EF0xeFgLDw1ZlVTNMXQqWPSODGFg8ezg2ltzcMpzGgB1MceqMrMbpxRVVoa0gO1pcebmPaAg8Qblzx4OFcby/s3Zy7emnHOkhOelVRHbq4zlwKS87IvIt62gKo9K+Umxj8k+4mht7eMKR3Rg+mOC7TzvXQfaA/bWVNuCFu5lJY9sb+YDFzWbg/woe6d86DhxtHentCgMg1UN+O9dCTmKSyclIMCsug+azMO6nr53mAcC6PsgcPScuBzxEmg7rbzG7N2v5h4M+AbmAr8B4zez5u6wF+G5MmJ4laAqwEZgK/At5pZp3lPhc3CvT2hGlFM+aBiKWJnc9BT+JjUDU5BoVmaDk3s5tr3Vyo8ClynTtUZQ0ekiqBO4BzCPOZr5a0ysyeTiR7Amg1s32S3gfcBrwjbttvZiflOPTfA581s5WS7gTeC3y+bCfihldPd3+AyG6H2PkcJGdhrJoSAkLDKxKD9cUSRN0cDxDOlUm5Sx6nAmvMbB2ApJXACqAveJjZjxLpHwOuGuqAcd7yN9E/F/q9wN/gwWNs6emG3S8kButLVDXtfD4zQFTXhmDQeHQYrC/ZzbVuDviMi84Nu3IHj3nAhsTyRuC0IdK/F/heYnmypDZCldatZvZtYBawy8y6E8ecV7osu5Lp6YJdL+S+k3rXC9Db3Z+2ujZ0a206Fo6+KLOKKdXkAcK5UabcwSPXNz7n5N+SrgJagTMTqxea2WZJS4EfSvotsCfH7oMd8xrgGoCFCxcWk29XqO7OGCBy3Em96wWwnv60NXWhkXrOiQMH60s1eoBwbgwpd/DYCCxILM8HNmcnknQ28FfAmWbW1yXGzDbHv+skPQKcDHwTmC6pKpY+ch4z7ncXcBdAa2trzgDjCtB9MFQl5ZoLYvcGsN7+tJPqQzCYe3LWYH3LoHa2BwjnxolyB4/VQEvsHbUJuIz+tgoAJJ0MfAFYbmZbEutnAPvM7KCk2cBrgdvMzCT9CLiE0OPq3cC/lfk8xr+uA7Dr+RwD9a2D3RuzAsS0UMU0/1Vwwjsyu7lOneUBwrkJoKzBw8y6JV0PPEToqnuPmT0l6RagzcxWAf8ApIAHQ1t4X5fco4EvSOoFKghtHumG9o8DKyV9itBb64vlPI9xo+tAuCEu153UuzeQUfs3eXr/QH0nXp45WN/UmR4gnJvgZDYxanNaW1utra1tpLNRfl37M+99SM4FsWcTGQFiyszMhunkUBtTZ47YKTjnRg9Jj5tZa/Z6v8N8LOrcF0oQue6k3rMpM+3UWVkD9S3rH25jyoyRyb9zbszz4DFadXbkuElufShN7H0xM+3U2SEoLDkjVi8t6S9BTJk+Mvl3zo1rHjxG0sG9/QEhWb20Y10YBjyptjEO1PemzIH6Zi6BydNGJv/OuQnLg0e5HdiTY5C+dIB4OTNtqikO1Hd2CArpdogZS2By/cjk3znncvDgUQoHdg8sOaQbrDu2ZqatmxMCQss5mT2YZi6FSamRyb9zzhXJg0eh9u/KPVDfjnWwb1tm2rq5IRj0DdSXqGKqqR2Z/DvnXAl58MjjpavfwMHnXgzjNCVVTQpDflc3QfWiMLpr9eTwV+mRXHcTbkN5Yphz7ZxzwaSjj+KIv/zLkh/Xg0c+k+thaidUT4nBIv7tCxDOOTfxePDI44g7V410FpxzbtTxn8/OOeeK5sHDOedc0Tx4OOecK5oHD+ecc0Xz4OGcc65oHjycc84VzYOHc865onnwcM45V7QJM5OgpK3A84e4+2xgW95U44uf88Qw0c55op0vHP45LzKzhuyVEyZ4HA5JbbmmYRzP/Jwnhol2zhPtfKF85+zVVs4554rmwcM551zRPHgU5q6RzsAI8HOeGCbaOU+084UynbO3eTjnnCualzycc84VzYOHc865onnwSJC0XNIzktZI+kSO7ZMkfT1u/4WkxcOfy9Iq4Jw/LOlpSb+R9ANJi0Yin6WS73wT6S6RZJLGfLfOQs5Z0qXxfX5K0v3DncdSK+BzvVDSjyQ9ET/b549EPktJ0j2Stkj63SDbJemf4mvyG0mnHNYTmpk/QrtPJbAWWArUAL8GjslK837gzvj/ZcDXRzrfw3DObwSmxv/fN5bPuZDzjenqgJ8AjwGtI53vYXiPW4AngBlxuXGk8z0M53wX8L74/zHAcyOd7xKc9xnAKcDvBtl+PvA9QMDpwC8O5/m85NHvVGCNma0zs05gJbAiK80K4N74/zeAsyRpGPNYannP2cx+ZGb74uJjwPxhzmMpFfIeA/wtcBtwYDgzVyaFnPN/A+4ws50AZrZlmPNYaoWcswH18f9pwOZhzF9ZmNlPgB1DJFkBfMWCx4DpkuYc6vN58Og3D9iQWN4Y1+VMY2bdwG5g1rDkrjwKOeek9xJ+uYxVec9X0snAAjP7znBmrIwKeY+PBI6U9KikxyQtH7bclUch5/w3wFWSNgL/Afz34cnaiCr2+z6kqsPOzviRqwSR3Y+5kDRjScHnI+kqoBU4s6w5Kq8hz1dSBfBZ4OrhytAwKOQ9riJUXb2BULL8L0nHmdmuMuetXAo558uBL5vZ7ZJeDfxrPOfe8mdvxJT0+uUlj34bgQWJ5fkMLMr2pZFURSjuDlVMHO0KOWcknQ38FXCRmR0cpryVQ77zrQOOAx6R9ByhXnjVGG80L/Rz/W9m1mVm64FnCMFkrCrknN8LPABgZj8HJhMGEBzPCvq+F8qDR7/VQIukJZJqCA3iq7LSrALeHf+/BPihxZaoMSrvOcdqnC8QAsdYrwsf8nzNbLeZzTazxWa2mNDGc5GZtY1MdkuikM/1twkdI5A0m1CNtW5Yc1lahZzzC8BZAJKOJgSPrcOay+G3CnhX7HV1OrDbzF481IN5tVVkZt2SrgceIvTWuMfMnpJ0C9BmZquALxKKt2sIJY7LRi7Hh6/Ac/4HIAU8GPsGvGBmF41Ypg9Dgec7rhR4zg8B50p6GugBPmZm20cu14enwHP+CPAvkj5EqLq5eoz/EETS1whVj7NjW87/AKoBzOxOQtvO+cAaYB/wp4f1fGP89XLOOTcCvNrKOedc0Tx4OOecK5oHD+ecc0Xz4OGcc65oHjycc84VzYOHc2OApDdIGi9DprhxwIOHc865onnwcK6EJF0l6ZeSnpT0BUmVktol3S7pV3FOlIaY9qQ4EOFvJH1L0oy4vlnSw5J+HfdZFg+fkvQNSX+QdN8YH9HZjXEePJwrkTjMxTuA15rZSYS7ta8EaoFfmdkpwI8Jd/4CfAX4uJmdAPw2sf4+whDpJwKvAdJDSJwMfJAw/8RS4LVlPynnBuHDkzhXOmcBrwRWx0LBFGAL0At8Pab5KvB/JU0DppvZj+P6ewlDwNQB88zsWwBmdgAgHu+XZrYxLj8JLAZ+Wv7Tcm4gDx7OlY6Ae83sxoyV0s1Z6YYaE2ioqqjkiMY9+PfXjSCvtnKudH4AXCKpEUDSzDjnewVhFGaAK4CfmtluYKek18f17wR+bGZ7gI2SLo7HmCRp6rCehXMF8F8uzpWImT0t6Sbg+3FiqS7gOqADOFbS44TZJ98Rd3k3cGcMDuvoH+X0ncAX4iiwXcDbh/E0nCuIj6rrXJlJajez1Ejnw7lS8mor55xzRfOSh3POuaJ5ycM551zRPHg455wrmgcP55xzRfPg4ZxzrmgePJxzzhXt/wPMli89qzup8gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on Validation Data: 0.438\n"
     ]
    }
   ],
   "source": [
    "num_train_epochs = 2 # How many epochs to train each network. Small value used just for quick demo, you should increase it.\n",
    "num_hpo_trials = 4 # How many networks to train with different hyperparameters. Small value used just for quick demo, you should increase it.\n",
    "ngpus_per_trial = 1 # How many GPUs to use per training run. Set =0 if you don't have GPU but beware of slow run-times (otherwise make sure GPU-version of MXNet is installed).\n",
    "\n",
    "classifier = task.fit(dataset, net=net, optimizer=optimizer, verbose=True, \n",
    "                      search_strategy='skopt', search_options={'base_estimator': 'RF', 'acq_func': 'EI'},\n",
    "                      epochs=num_train_epochs, num_trials=num_hpo_trials, ngpus_per_trial=ngpus_per_trial)\n",
    "\n",
    "print('Accuracy on Validation Data: %.3f' % classifier.results['best_reward'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We again evaluate the resulting model on our (small) test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73370e6f876348169b004cdb76890ce0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, max=1), HTML(value=''))), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy on test data: 0.531\n"
     ]
    }
   ],
   "source": [
    "test_dataset = task.Dataset('data/test', train=False)\n",
    "\n",
    "test_acc = classifier.evaluate(test_dataset)\n",
    "print('Accuracy on test data: %.3f' % test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperband\n",
    "\n",
    "AutoGluon can either schedule trials in serial order or with preemptive early termination. \n",
    "In standard Bayesian optimization, we wait until a training job completes (the network has completed all of its training epochs), see what the resulting validation performance of the fully-trained network is, and then use this information to decide which hyperparameter-values to try next. However, we can also measure the validation-performance of the network after each epoch *within* a training job. If a network displays poor performance in the first few epochs, this is likely not a promising hyperparameter-configuration to explore further, and thus the trial may be terminated early to free up compute resources to start training a different network.\n",
    "\n",
    "\n",
    "Here's an example using an early stopping scheduler that implements the [Hyperband search strategy](https://homes.cs.washington.edu/~jamieson/hyperband.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "scheduler_options: Key 'reduction_factor': Imputing default value 3\n",
      "scheduler_options: Key 'brackets': Imputing default value 1\n",
      "scheduler_options: Key 'type': Imputing default value stopping\n",
      "scheduler_options: Key 'keep_size_ratios': Imputing default value False\n",
      "scheduler_options: Key 'maxt_pending': Imputing default value False\n",
      "scheduler_options: Key 'searcher_data': Imputing default value rungs\n",
      "scheduler_options: Key 'do_snapshots': Imputing default value False\n",
      "\n",
      "scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60\n",
      "scheduler_options: Key 'delay_get_config': Imputing default value True\n",
      "\n",
      "Starting Experiments\n",
      "Num of Finished Tasks is 0\n",
      "Num of Pending Tasks is 16\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheduler: HyperbandScheduler(terminator: HyperbandStopping_Manager(reward_attr: classification_reward, time_attr: epoch, reduction_factor: 3, max_t: 10, brackets: [Bracket: Iter 9.000: None | Iter 3.000: None | Iter 1.000: None])\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f631b105dd7f43dc8199351736d19d8c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=16.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving Training Curve in checkpoint/plot_training_curves.png\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOydd3hVRfrHP296IwkkJCGBJLSEXkORItIRqVZALGvva9fdn7qrq65rL+vauwh2AUFpSpEaepME0nslvd87vz/OJcaQkNyQmwLzeZ48ueecmTnvafOd9s6IUgqNRqPRnL/YtbYBGo1Go2ldtBBoNBrNeY4WAo1GoznP0UKg0Wg05zlaCDQajeY8RwuBRqPRnOdoIWgBRORpEckWkfTWtuV8RUTsRaRIRIKbM2xzIyIfi8jDLXCeKSJywNbnaSwi4iIiSkS6trYtzYmIbBaRyxoZNltERtvaprrQQlAHIhIvIqWWzCBDRD4SEY8mptUNeADop5QKaF5Lz10s9/7Un7nG8ygSkautTU8pZVJKeSilEpszrDWISFit61IiUlxje4RS6nql1PPNed66UEqtV0oNtvV5bIWIPCci79fa9ycxsYSptNzbPBH5TURG1AgfIiLLRCTXEmaHiEyv53xTajynYst5aj5Lv7riKaUuVEp925zXbgu0ENTPbKWUBzAMGAE8Zm0CIuIAhAA5SqnMJsY/L7FkxB6WZ5CI5XlY/pbUDt8e7pVSKrrGNfladofXuK7I1rTvHOUTy/32A/YAXwNYMu6tQB7Qx3L8LeAbEZldOxGLcJ56dsMBU813tPb3LSJ2IiI2vbJmRAtBAyilUoCfgAEAIuIlIh+ISJqIpFiafewtx64Xka0i8oqI5AIbgXVAoKXU8LEl3BwROWIppWwUkb6nzmepjTwiIgeBYhFxsOx7SEQOWkojH4iIv4j8JCKFIrJeRDrWSONrEUkXkXxL1bR/jWMfi8ibIrLKEneniPSscby/iKyzlJIyROTvlv12IvKoiMSISI6IfCUineq6ZyLyu4jMqrHtYKn2DrOU2j63pJEnIpEi4m/tc7Hc9y9FZKmIFAKLReQCS6kuz/J8XhcRxxo2KBEJtWx/bjl+6h5uF5Hu1oa1HL9YRKIt9/sNyztwvbXXZElrmYg8Zvk9Q0ROiMhjlvuXIiIzRWRujefwQI249iLyuIjEWsIvERHves4zQ0RO1NhOF5H7ROSw5TqWiIjTGey8VUSiLO/JKhEJqnHsLRFJFpECEdklNZo7LPf2HxYbCyzPv2ZN+WLLtZ0UkVeacg/rQilVAXwKhIhRu38IyFBK3aaUylRKlSilPgFeBF5qyjlEZLeI/FNEdgElgL9l3wLL8X6W7zFXRDLlLFoamhstBA0gRtPOTGCfZdcnQBXQCxgKTANuqhFlFBCLUcKYClwMpFpKDdeLSBiwFLgX6AysBlbW+ugWApcA3kqpKsu+yyzphQGzMcTp7xglSzvgnhrxfwJ6W2zYC9QuQS8EngQ6AieAZyzX2gFYD/wMBFqucYMlzj3APGCC5dhJ4M16bttSyzlOMR3IVkrtBa4DvIBugA9wG1BaTzoNMR/4wpLelxjP5a8Y92QsMAO49QzxFwGPA50wah3/sjasGCXLrzAyFl8gDhjZxOupixCgEggAngM+BC4HBgFTgGdqZMIPYbyP44CulnjWZKaXA5MxnvsojGs+DUvGdi/Ge+iP8W18XiPIdmAgxvNdDnx9SpCBv2G8R9MAb+AWoKxG3IsxvqthwF9E5CIr7K8XEXHBePdOKKWKML6lb+oI+hXQW0RCmniqxcDVgCeQVcfxJzDu2WCgP/BoE8/TvCil9F+tPyAeKMKoNiYA/wNcMR5gOeBaI+xC4FfL7+uBxFppXQQk19h+HPiqxrYdkAJcVOPcN9Rhz9U1tr8F3qqxfTfwQz3X4g0owMuy/THwfo3jM4FjNa5lXz3p/A5MrrHdBSOjcagjbC+gEHCzbC8BnrD8vgHYBgyy8nlMqbXvaeCXBuI9CHxt+e1guQ+hlu3PgbdrhJ0DHG5C2BuALTWOCZAGXN+AbS6Wc3SttX8Z8Jjl9wwgH7CzbHe2xBlcI/wRYIbldxwwtsax7hglU6nj/DMwMsVT2+nA5TW2Xwdercf2X2u9j46Wd8G/jrBisSHcsp0ATD/D/YiosW8FcG89NjyH8S3m1fqrvqe1wmRi1M4HWY4l1/WM+ON7GX6GZ9cHqKpj/27g4Tr2LagnncW13p1sYHRjv4vm/Gvz7aqtyDyl1PqaO0RkIMZLnyZ/NP/ZAUk1gtX8XReBGB8DAEops4gkAUE1wtSVRkaN36V1bHtYbLTHKOFfgZFxmC1hfDEyFTA++lOUnIqLUUqPqcfuEOB7ETHX2GfCEMeUmgGVUidE5HdgtoisxMg4h1oOf2Y5zzJLs8XnwP8ppSrrOe+Z+NN9EpE+GNX64YAbRoa+8wzx67sP1oQNrGmHUkqJSHINm6L449lOVUptP8M56iJLKXXqnp+qOZ327MV4IbsBq0Wk5kySdhgl8+xGnKv2NfrWEy4EeFtEatYIqzBqIRki8jeMQlEARqbqAviKSDTGvajvHavLhjM9k8+UUtW1cUupv3bt8k9hapCNUZipTZcax5tCvd+/GJ3YrwJjMK6rdt7RauimIetIwihh+CqlvC1/nkqp/jXCNDSdayrGhwRAjQ+4ZmZ6NlPCLgLmYjQbeAGhp07ViLhJQM8zHLu4xnV7K6VclNGHUhenmofmAkeVUicAlFKVSqknlVL9MD6IWcC1jbCtLmrfp3eAw0AvpZQnRjXc1h12aRgZIFD9PKtFXSlVszPYWhFoNMooUqYAk+p4Rk3N1OojCaM0XfM8rkqpPSIyFaOGOh+jdN0JI3OWGjbW9461JOsxmsJqcyUQo5RKqONYYzjTt/sSRmGsn+X9vA3bv5+NQguBFSil0oC1wEsi4ilGB2pPEZlgRTJfAZeIyGRLu+kDGOKyrZnM7GBJLwejVPysFXF/BAJE5F4RcRaRDiIyynLsbYz26BAAEeksInPPkNYyjHbg2zHa8bHEmygiAy01lwKMJgWTFTaeiQ4YH1qxGB3wZ+ofaC5+BIaJyGwxRi79FaMm1hq8DTxn6ddCRPykjhEwzXSex0Qk3HKejvLHWPkOGM80C3ACnsKoEZzifeBZEekhBkPr69C2MS8AXUTkbct9chWRazH6WR600Tk7YDSZFogxEOE+G53HarQQWM+1GC/4UYwO02+ou4pZJ0qpKIy2wTcwqp+zMYZGVjSTfZ9iND2lWGzcYYVthRidaLMxqujHgYmWw69htNmuFWOUzg6MDsX60krD6DQcg9GRe4oAjHtWgNHvsIk/dzSeDQ9gdAgWYtQOvjxz8LNHKZUBXAW8jCG+PTE6T8ttfe46eB6jpPuL5Rltw+h0bVaUUkuB/wLfiUgBsB/jvQFYCWzGaP6JxXjHa3aaPgesAn7BeAfeBpyb28aGsDy3sRg1ligMO+8ErlRK/WCj0z6G8T0VYHwDX9voPFYjlk4KjUbTDFhqOqkYHa9bWtsejaYx6BqBRnOWWMbke4mIM8aosCpgVyubpdE0Gi0EGs3ZM44/mkFmYIw4a42mIY2mSeimIY1GoznP0TUCjUajOc9plw5lvr6+KjQ0tLXN0Gg0mnbFnj17spVSpw1vbpdCEBoayu7du1vbDI1Go2lXiEidjnK6aUij0WjOc7QQaDQazXmOFgKNRqM5z9FCoNFoNOc5Wgg0Go3mPEcLgUaj0ZznaCHQaDSa8xwtBBqNps2Slb2BrKy1/LFIm8YWtEuHMo1Gc+6TnPIFUVGPA+Dm1oOQ4FsICJiDnV2LL19wzqNrBBqNps2RnPw5UVGP4+szif79X8XezpXfjz3Ktm0TSUh8n6qqotY28ZxC1wg0Gk2bIin5M6Kj/4mv7xQGDngdOztn/P1mkZv7GwkJb3PixL+Jj3+TrkGL6dbtOpycfFvb5HaPFgKNRtNmSEr+lOjoJy0i8AZ2dk4AiAg+PuPx8RlPfsEBEhLeIT7hLRKTPiCwy5UEB9+Iq2u3Vra+/dIu1yOIiIhQetI5jebcIinpY6KP/4vOvlMZMOD1ahGoj+LiGBIS3yM9/QfAjJ/fJYSE3EoHjz4tY3A7RET2KKUiTtuvhUCj0bQ2iUkfcfz403TuPI0B/V/Hzs6x0XHLytJISvqIlNSlmEwl+PhcREjIbXh7RSAiNrS6/aGFQKPRtEkSEz/k+Iln6Nx5BgP6v2qVCNSksjKP5OTPSUr+hMrKXLw8hxISchu+vpMQ0eNiQAuBRqNpgyQmfsDxE8/i1/li+vd/pckiUBOTqZTUtG9ITHyPsrIU3N17ExJ8M/7+c5ol/fZMfUKgZVKj0bQKCYnvGSLgN7PZRADA3t6Vbl2v4YLRG+jf72UEO47+/jDbtk8kMekjTKaSZjnPuYSuEWjOe5QykZOzmZzcLbi798LbKwJ39166OcGGJCS8w4mY5/Hzu4T+/V7Gzs52AxiVUuTkbCQh4R3y8iNxcPCmW9dr6dr1GpycOtnsvM2BUoqKikyKiqIoKo6iqCiKkOCb8fAIb1J69dUI9PBRzXlLRUU2qalfk5K6jLKyZEQcUaoSAAcHL7y8huHtFYG3dwQdOgzE3l57tDYH8QnvEBPzPP5+s+jX7yWbigAYQ099fSfi6zuRvPw9JCS8S1z86yQkvkdQ4FUEB9+Ii0ugTW1oDFVVhRQVR1NUFEVxUXT176qqvOowTk6d8fe7pMlCUB+6RqA5r1BKkZcXSUrKEjKz1qBUJR29RxPU9Wo6+06hvDydvLzd5OXvJi9vDyUlJwAQccLTcyDeXhF4eQ/H22sYjo4dW/lq2h/x8W8RE/si/v5z6Nf3BZuLQH0UFUWTkPguGRkrAfD3n01I8C14eITZ/NxmcyUlJbGWUn40xZbSfllZSnUYe3t33N3D8PAIw8M9HA+PcNzdw866BqM7izXnNVVVhaSlfUdK6lKKi4/j4OBJly6XERS4EHf3nvXGq6jIJT9/L3n5u8nP201B4eHqWoO7e2+8vIZX1xpcXLrq4YpnID7+f8TEvkSA/1z69n2+1USgJmVlqSQmfkBK6peYzaX4+k4mJORWvL2Gn3XaSinKylIptjTpFBUbJf3iktjqd0jEATe37ni4h+NeI9N3cQmySdOkFgLNeUlBwSFSUr4gPWMlZnMpnp6DCQpchL//Jdjbu1qdnslURkHhIfIttYb8/D1UVRUC4Ozkb6ktDMfLOwIP9z5tIrNrCicid5CbmsyI2ZcidmefIcXF/ZfYuFcI8J9Hv37PI2LfDFY2HxUVuSQnf0ZS8qdUVeXh7TWCkJBb8fG5qFHiXlmZR1FRtCWzP9WeH43J9MecSM7OXfDwCP+jhO8Rjrtb9xadRE8Lgea8wWQqJSNjFSkpSygoPIidnSsB/rMJClqEp+fAZj2XUmaKi4+Tl2eIQl7+7uoqvr29O16eQ/HyjsDbaxienkNwcHBv1vPbgpRjR/nqqb9jNlUxeOrFTL7h9rMSg9i4N4iLe5WAgHn069v2RKAmJlMJKalfkpj4AeXlaXi4hxMScit+fpdgZ+eAyVROScmJP3XeFhdFU16RUZ2Gg4OnpYQfbsn4w/DwCMfBoUMrXplBqwmBiMwAXgPsgfeVUs/VOh4MfAJ4W8I8qpRafaY0tRBo6qK4OIaUlC9IS/+OqqoC3N17ExS4kICA+Tg6eraYHWVlqeTl7yE/zxCGoqJjgELEHg+Pfnh7Rxh9DV7DcXbu3GJ2NYaC7CyW/P0+nFxd6TFsJHtXL2fQ5BlMuemOJolBbNzrxMW9RpeAS+nb97k2LQI1MZsryMhYSULiexQXH8fFORA7e1dKS+NRygQY/Ubu7r0s7fhhRsbvHo6zc0CbbSJslVFDYjz1N4GpQDIQKSIrlFJHawR7DPhKKfWWiPQDVgOhtrRLc+5gNleQlb2elOQlnMzbgYgjfp2nExR0Nd7eI1rlg3RxCSTAJZAA/9mA0T9h9DPsIS9vNykpS0lK+ggAV9eQ6qYkb68I3Nx6tFomUllexvIXn6aqopwrn/g3nYK64ujszM7vv0KhmHrTnY0WA6UUcXGvERf/Bl26XE7fPs+2GxEAsLNzokuXywgImE929i8kp3yOnZ0zfn4zqpt2XF1D223TX21sfRUjgRNKqVgAEVkGzAVqCoECThXXvIBUG9ukOQcoK0slJWUpqWlfUVGRjYtLV3r2eIjAwMvb3LTEDg4d8PGZgI/PBMAQr8LCo9Ud0Nk5G0lL/w4AR8dOdOw4mm5dr8Pb+7SCm81QSrH2nTfIjI9l3kOP4dPVmMlz7FXXAMLO779EmRXTbrmrQTFQShEb9yrx8f+lS5crLCLQPn0yROzo3HkKnTtPaW1TbIqthSAISKqxnQyMqhXmn8BaEbkbcAfqvOMicgtwC0BwcHCzG6pp+5xy/EpJ+YLsnI2AwtdnIkFBi/DxubDdlDjt7Jzw8hqCl9cQCL4JpRSlpfHVw1azszeQmbkaL68IQkNua3SH5dkQueJbjm3dxLgF19Jz+B+fqIgw9qrFiJ2w49tlgGLaLXfXKwaGCLxCfPybBHa5kj59nmm3InA+YWshqOvtrd0psRD4WCn1kohcAHwmIgNUrUVKlVLvAu+C0UdgE2s1bZLajl9OTr6EhtxGYOACXF2DWtu8s0ZEcHPrjptbdwIDr8BkKiE19WsSE9/nwMGbcHcPIyTkVvz9LrHJXDmx+yLZsvQTwi8Yz8h5V9Rp39grFyMibP9mqVEzuO1u7Oz+LLxKKWJjXyI+4S0CA6+iT/jTWgTaCbYWgmSg5moRXTm96edGYAaAUmq7iLgAvkCmjW3TtGHqc/zq1fNhOnee2uBc9e0Ze3s3unW7jqCgRWRk/EhC4jscPfoAsbEvE9ztRgIDr2zS0Ne6yElJYtVrL+AX0oPpt//1jDWPMVdcDQjbv/kCUEy77Z5qMVBKERP7EgkJbxEYuIA+4f/SItCOsLUQRAK9RaQ7kAIsABbVCpMITAY+FpG+gAuQZWO7NG2Uuhy/ugZdTVDQojM6fp2L2Nk50qXLfAIC5pKTs5H4hLeJPv4UcfH/rZ4rx9HRu8nplxUXsfyFp7F3dGTuQ/+Ho7NLg3HGXLEIEWHb10tQZjPT77gXETtiYl4gIfEdgoIWER72pBaBdoZNhUApVSUidwFrMIaGfqiUOiIiTwG7lVIrgAeA90TkPoxmo+tVe3Ru0JwVpzl+dRhE3z7/abLj17mEiB2+vpPw9Z1EXt5uEhLeITbuVRIS3yUwcAHB3W7AxaWLVWmazSZWvf4C+ZnpXPH4M3j6+jU67gWXLwSBbV8tQSlFr+mVJCW/T1DQ1YSH/VOLQDtEO5RpWo3y8iwyM1eRlv4DhYWHsLNzIcB/jk0cv841ioqiSEh4l4zMlYAdAf5zCAm5BXf3Xo2Kv3nJR0Su+JYpN93J4KkXN8mGHd8tY+uXn9OxVz5jFk8ivM8/2+z4eY2Bnn1U0yaoqiokM3MNGRkryT25DTDj4dGXsN5PtLjjV3vGwyOc/v1fokeP+0hM+oDU1K9IS/+Wzr5TCQm5FS+vofXG/X3Lr0Su+JbBU2c2WQSUUvgMjKVLXCZpu/yIWedBWJgZsW8fI7c0f0YLgcbmmExl5ORsJD1jBTk5v2I2V+DqEkxo6O34+8/Gw713a5vYbnF17Up42D/oHnoXScmfkpz8GVnZ6/D2HkVoyK106nThn0rp6THHWfvOG3TtO4CJ19/cpHMqpTh+4hmSkj5i+OzryOvZh9+WfopSipl3PYCdlWKglKKgtIqUvFJS80pJzS8lJa+UtLwy0vJL6e3fgbsm9iLQ+/xuIrQlumlIYxPM5ipOntxORsYKMrPWYjIV4eTki7/fLPwD5uDZYZBuRrABVVXFpKZ+SWLSB5SXp+Ph0ZeQ4Fvw85tJaUEhn//tXsTOjsX/fhU3Ty+r01dKcfz40yQlf0y3rtfTu/djiAi7ln/Dli8+JvyC8cy8+8E/iUFFlZn0/DJS8y0ZfV4pKXll1b9T80oprjD96TyO9kIXL1f8OjhzMDkfBK4ZHcIdF/XEx0OvC9FU9KRzGpujlKKgYD/pGSvIyFhFZWUO9vYe+PnNIMB/Dh07jj7N6Sv56GG2f7eMDj6+9Bg2gpCBQ3F2c2ulKzh3MJsrSM9YQULCu5SUxODk0I2YH0PJTy9k4VMv4Bfaw+o0lVJEH/8Xycmf0K3bDfTu9XcATpZUkppXyv7V35Ox/htMIYNIGDiflIIKUvNKySoqp3Y24+PuRKC3K4HeLgR6uxLk7UoXL2M7yNsVXw9n7OyMgkLyyRJeXX+c7/Ym4+poz43je3Dz+O50cDm/1x9uCloINDajqCiajIwVpGf8SFlZEnZ2Tvj6TMY/YDY+nS6qc2Wv8pJiNi/5iIPrf8ajYycqK8opLy7Gzt6eoD796TE0gu7DRtApUM/xfzYoZSYrax3r3n2T9ENmes7IZ8jEhQQFLW5Uf0xZpYm0/DJSTpaQl/4fXCu/J750DhvTr6ou1ZdV/uH7OTR/P+Nyt5PaMYz0YZcR2NGjOqM/lfF38XLF1cn6voQTmYW8tDaanw6n09HNkTsu6sU1F4Tg4mj7fgllNqNQpznRtTe0EGialdLSFDIyfyQjY4Vldk07OnUaS4D/bDp3nnbGKXdPRO5gwwf/ozgvj2Ez5zD2ysXYOzqSGv07sft2E7c3kuykBAC8/APoMXQE3YdG0K3fQByczl1HMlux96eV/PrxOwyeOQ7fwVHk5G7G3t6DoCBj6Kmzs391WKUU22JyWLIzgV1xuWQXVQCKq/t8zaTg3/g5fhIb064k0NuNIG8XAr1OZfCnMnsX4jauZtNnH9B71Bguuedh7B2atyvyYHIeL6yJYsvxbAI8Xbhncm+uiOiKo33zD1tVShG7dxe/LfuMYTPnMHDitGY/R0uihUBz1lRU5JCZ+RPpGSvIz98DgJfnUPwD5uDnNxPnBiZ7K847yS8fvUP0jt/wDQ5l2q1306VX3WuvFmRlGqKwL5LEwwepqijHwdmZ4AGDq4XB07dtTeHcFkk8fIBvnnmcHsNGMPeB/0Ps7CgsPGpZpnEVIg50CZhHR7/r+fGoI1/sTCQ2uxhvN0em9vWnWycXejq/g0vFcrw7X0//Pn/DxbHhjH3PquVs/PQ9eo24gFn3Poy9Q/M342yPyeH5NcfYl5hHqI8b900NY/agwOompbMl6chBtiz7lLToY3gHdOGia2/60zxMrYHJrLATmlxL1kKgaRJVVUVkZa0jI3Mlubm/oZQJd/feBPjPxd9/Fq6u3RpMQynFkY3r2fTZB1SWlzH6soWMmHNZo0uKlRXlJB85ROy+SGL37qYgy1gExDc4tLoJKbB3H6tHq5zr5GWks+Tv9+Hu3ZGF/3rxtL6XkpIE9h79HyX5yxGq2Js5iLjS+UwbMpGZA7vg7CBERf+DlJQvCAm+lZ49H7IqA9q7ejm/fvIevUaMZta9j9hEDJRSbPg9kxfXRnEsvZC+XTx5aHoYE8P9mpxZZsSeYMvST0g4uA+PTj5ccNlC+l80pdlrNo1FKcWhlHyW709l5YFU3lo8jOEhTVu7WAuBptGYzeXk5GwmPWMF2dm/YDaX4eIShL//bAL85+DhUXcpvi7yMtJZ9+4bJB4+QFCffky95W58ghoWj/pQSpGbkkzsvkji9kaSEnUUs8mEi7sHoUOG02NoBCGDhzVpRMy5REVZKUsfe5Ci3BwWPfsyHQMCq48Vl1exfH8qS3YmcCS1gAD3Ym4evofubj+jzEV07DiGkJBbycz8idTUZYSE3E7PHg80KWM91SzVM2I0s++zjRgAmM2KlQdTeXldNAk5JUSEdOSh6eGM6uHT6DRykpPY+tVnHN+5DZcOnoyaezmDp1+Co1PrjFKKySqqzvzjsotxtBcuCvfj7km9GNS1aVOLaCHQnBGlTJw8uZOMjJVkZv1MVVUBjo6d8Pe7BP+A2Xh5DrMqIzCbTOxdvZytXy3Bzt6O8Yv+wuApM5pl/dualJcUE39gH3H7dhO3fzcl+XmI2BHQO4weQ0fQY9gIOod0P686nJXZzIqX/03M7p1c+vcnCR1kOJdFpReyZGcC3+1Noai8ij4BHbh6dAjzhwbh4exAVVUhKSlLSUz6iIoKY87H0JDb6dFEETjFvjU/8suHb9Nj+Ehm3/c3HBxtN9qn0mTmy8gkXt9wnMzCci4M68zD08MZEFR/waAgK5Nt33zB0U2/4ODsTMSseQy/ZH6rjF5Lyy/lxwNpLD+QwuGUAkTggh4+zBkcyMUDuuDldnb3TguB5jSUUhQWHiI9YyUZGT9SUZGJvb07nTtPI8B/Nh07jmnStMeZ8bGsfecNMmKP02P4SKbceAcdfGy/WIwym8mIPVHdhJQRexwAj46d6G5pQgoZOAQnl3PbMWnb11+w/ZsvuOjamxgwfTY/H07n8x0JRMafxMnBjksGdmHx6GCGBXesM4M3m8tJT1+OWVURFLiwWUR0/5pVbPjwLXoMG8Hs+/9uUzEAKK0w8en2eN7aFENeSSWXDOzC/dPC6NnZozpMSX4eO77/koPrfgIRhkybych5V7Z4bTKvpILVh9JZvj+FXfG5KAWDu3oxe3AgswcH4u/Z8GSAjUULgcay0PqJ6kXW8/IiKStLQcQJX58J+AfMwddnEvb2TXvxqioq2PHdMiJXfIuzuweT/nIr4ReMb7XSeHHeSeL27yFubyTxB/dRUVqCvYMDXfsNpPuQCHoMi6Bjl/a/nkFNju/cxoqXnyVk9ASOh1/C13tSyC2uIMTHjatHBXP58G50cm+dkVf7165mwwf/azExACgoq+T9zbG8/1scZZUmLh/eldvHBJGy+Sf2rl5OVWUFAy6awujLFrbo4IOSiirWHc1g5YFUNkVnUWlS9OjsztzBQcwZEkh3X3ebnFcLwXmI2VxOQcEhy0Lqu8nL30NVVT4ATk6+eHlF4Oszgc6dZ5z1HD/JRw+z9t03OJmWQv8JU5hwzQ24dmg78waZqqpIOXbU6FvYt5vcFGPhvI5dAuk+dAQ9ho4gqG//FsmcbEVGfBxfPPYghW6+fNpxJsrekWok6CIAACAASURBVCl9/bh6VAjjevk222ias+HAup9Y//6bdB8awZz7/95iw4Gzi8p5c91RDq9bzZCTe3ExlxM6YiwTF11Lp8CWKQxUmsxsOZ7F8v2prD2SQWmliQBPF+YMCWTO4ED6B3ravNCkheA8oLIyz1gk3bLkYWHhIczmCgDc3Hri7TUcb+8IvLyG4+oa0iwvXXlJMVu++JgD637Cs7M/U2+5q7pNui2Tl5FO3L5IYvftJunIQUyVlTi5ujFs5hwiZs3H2c02JTJbkFlQxtItv5O/7AWUqYpfwhcxb0w/FozsRhevttcMdnD9z6x777+EDhnO3Af+z+ZiYKqq5NAv69jx3TKKT+ZSHhDGD/aDKPYI4MZx3bnpwh542shL2WxWRMbnsvxAKqsPpZFXUom3myMXD+jC3CGBjAzt1KICrYXgHEMpRVlZcnWmn5+/h+Jio01cxBHPDgPw8h6Ot1cEXl7DcHJq/OiJxnJi907DMezkyWrHMEeX5mvPbCkqy8pIPHKQo5s2EL1zKy4eHRg593KGzJjVaiNGGkIpxfaYHD7fmcD6w6nMTF1JUEUGPa9/hNlTRtvEuao5ObhhDevefYPQwcOY++BjNhEDs9nEsa2b2fb1EvIz0gnq049xC66la98BnMgs4pV10aw6lIa3myO3T+jJdWNCm8VLWSnFkdQCVh5IZcWBVNLyy3B1tGdqP3/mDglkfO/OODm0zvPRQtDOMZurKC6O+iPjz9tDeYUxnt7BoQNeXsMsmX4Enp6DmtzO3xiscQw7E6UVJj7aFseyXUlEhHbkvilhdOvUuvMMZcSe4LcvPyN+/x48OnZi9GULGTBxaquNIa9NXkkF3+xJ/pPj10LTHhyObWXGHffRf8Lk1jax0Rz6ZS1r332DkIFDmPvQY80mukopYvbsYuuyT8lOSqBzaA/GL7iW0CHDT6sFH0rO58W1UWyKzsLf05m7J/XmqhHdmiSk8dnFrDiQyvL9KcRkFeNgJ0wI68ycIYFM7eePm1Prv0NaCNoZJlMJ+fn7q9v38wv2YTIVA+DiHIiXdwTeXhF4e0fg7t67RVaFqtsx7FKrx4ZXVJn5MjKR1385QVZhOcNDOnI4JR+zUiwaGcydk3rh16F1axbJRw+zZeknpEb/jrd/F8ZceTV9xlzY7MNfG4NSiv1JeSzZmcjKA6mUV5kZFuzN4tEhhOYc5tf3/8vwS+Zx0bU3tbhtZ8uhX9ey9p3mE4PEwwf5bdknpB2PomOXQMZcuZjw0eMafG47Y3N4fk0UexJOEuLjxv2N9FLOLChj5cE0VuxP4UCy0f82qnsn5gwJZOaALnRspY75+tBC0MYpL8+qHs2Tn7eHwqIjKGUCBA+PPtVNPN7eEbi4BDaYXnOTl5HOuvf+S+Kh/U12DDOZFcv3p/DK+miScksZGdqJh2aEMyK0E2n5pby+4QRf7U7Cyd6Ov4wN5dYLe571uOmzQSlF3L7d/LbsU7IS4vANDmXcgmvoMWxki4yEKqkwHL8+32E4frk72TNvaBBXjwqhX6AnKVG/89WTf6Nb/4Fc+ug/261n9eGN61nz9msEDxjMvIcea9TaybVJjznOb8s+NbyBfXwNb+AJk62qySml+DUqk+d/NryU+wR04MFp4Uzu+2cv5fzSStYcTmf5gRS2x+RgVjAgyJM5gwOZNSiwTa+boIWgDaGUoqQkjvz83dVNPaWlxiRrdnbOeHoOqdGxO+yME7jZmuZwDFNKsfZoBi+tjSI6o4j+gZ48ND2cCWGdT8tQ47KLeWVdNCsOpOLp4sCtE3ryl7GhrVqtVmYzUdu3sPWrz8lLTyMwrC/jFl5Lt362WU4zOqOQz3ck8P3eFArrcPwCKMjOYsnf78PJ1ZWrn3kFFw+PBlJt2xzZtIGf33qV4P4DmffwE40Wg5zkJLZ++RnHd23DtYMno+ZfyeCpM8+qz+GUl/Ir66KJzylhWLA3908NJ7+0kuX7U9gYlUWFyUyojxtzhgQxZ3Agvfzax/3XQtBKVFbmU1QURVFxFMWW/0VF0ZhMRQA4OnbC22t4dVNPhw79sLNrG9XJrIQ41rz9+lk5hm07kc3za6LYn5RHD193HpgWzsUDAhqsch9NLeDFtVH8ciwTXw9n7pnciwUjglutkw2MIahHNq5n+7dLKcrNIWTQUMYvvA7/Ho1bJ/hMlFeZ+PlwOkt2JLIrPhcnezsuGVS341dlRTlf/uMRTqalsOjpl/DpGnzW528LHN38Cz/97xW69RvI/IefOOPAg/zMDLZ/8wVHN/+Ko4szEbMuZdjMuc3qDVxpMvP17mRe2xBNRkE5AH4dnJk92BjuOairV7vzWNdCYGNMpnJKSk5YMv1oS6YfTXl5enUYBwdPPNzDcfcIx7NDf7y8InBza3vTHzSHY9j+pDxeXBPFbyey6eLlwr1TenPZsK44WNkJtzs+l+fXRLErLpeuHV25b0oY84YGYd+KY+IrK8o5sGYVO5d/Q1lhAb1HjWHsVdc0aQ6lpNwSluxM5OvdSeRYHL8WjQzmioi6Hb+UUqx+40WObdvMvIcea/XZMJubo1t+5ec3X6Fr3/7Mf+Qfp4lBcd5Jdn7/FQfW/YTYCUOmz2Lk3Mtt6g1cVmli1cE0uni5MKqHT6u+e2eLFoJmQikzpaVJFBdHVWf6RUVRlJbGW9r0QcQJd/deeHiEWTJ+47+zc0Cby/Rr82fHsMlMuOZGqxzDjmcU8uLaKNYcyaCTuxN3TuzF1aOCz2pYnlKKTdFZvLAmiiOpBYT5e/DAtHCm9fNv1ftZXlLCnlXfs/vHH6gqL6ffhEmMuXwRnp39zhjPZFb8ciyTJTsT2BSdhQBT+vqzeHTDjl+RK75l85KPGLfgWkbNv7KZr6ht8PuWX/npzVcI6tuPSx/5J44uLpQVF7F75XfsWb0cU2UlAydOY/RlC1pk6pJzCS0ETaCiIvtPmb1Ryj+O2VxaHcbVJdjI6D3C8XAPx8MjHFfXUOzsWn+omDWc5hh2852EDh7W6PhJuSW8sj6aH/al4ObkwM3je3Dj+O7VbdrNgdms+OlwOi+tjSI2u5jB3bx5eHo4Y3u1bmZQUpDPrh++Yv/a1aAUg6ZezOj5V+Hm9ecZIjMLy/hyVxJLdyWSml+Gv6czC0YEN9rxK27fbr77z5OEjR7HrL8+3OYLFWfD71s38dMbLxHUpx+hQ4aze8W3lBUX0WfsBMZcseicmxqkpWg1IRCRGcBrgD3wvlLquVrHXwEmWjbdAD+l1BnnWG1uITCZSigqPl7dnFNUZJT2KytzqsM4Onb6U2bv7h6Gu3tvHBzajwdqfZyNY1hmYRlv/nKCL3YlYifCdWNCuW1CT5vOZ1NlMvPt3mReW3+c1Pwyxvby4cFp4QwN7mizczaGguwsdny7lMMb1+Pg6MSwmXMZPmse+9LL+XxnAmuPZFBlVozv7cvVo4KZ3Ne/0ePVc1OT+eL/HsDTz5+FTz7fLh33rOXY1k2sfuMllDLTY9gIxl51TZPWWtb8QasIgRgrlUcDU4FkIBJYqJQ6Wk/4u4GhSqkbzpRuU4XAbK6itDTB0mH7R+dtaWkSYNwHOzsX3N17/znT9whvcPWt9khx3kl++fhdordvsdoxLL+kknc2x/DR1ngqTGauGtGNeyb1JsCr5TKoskoTS3Ym8uavJ8gtrmBaP38enB5OmH/rjbICyE1NYdPSz4jd9RsV9i5Eeg4hIWAol47ozqJRIVZPKFZWXMQX//cAZcVFLH72lQabns4lko8dwc7OnsCwPq1tyjlBfULQYL1dRM7YPqCU2nuGwyOBE0qpWEtay4C5QJ1CACwE/tGQTU0l+vhTpKQssWzZ4eYWSgeP/gQEXIqHexgeHmG4ugZj6Ne5S2FuNntXr+DAup8wV1Uy9qprGu0YVlJRxcfb4nl7YwwFZVXMGRzI/VPDCLXRbIlnwsXRnhvHdeeqEd348Lc43tscy/RXNzN/SBD3TW15L2WlFAeS8/l8RzYrTw6mQ2AA08v2MjZ3B1NVFBc4LiTYu7dVaZrNJla//gL5melc8dgz55UIAHTt07+1TTgvaLBGICK/Wn66ABHAAUCAQcBOpdS4M8S9HJihlLrJsn0NMEopdVcdYUOAHUBXdarX9c/HbwFuAQgODh6ekJDQ8NXVIi/PGK/v7hGOu1svm07D0BbJTU0mcsV3HN38C0qZCb9gPBdcvpBOgV0bjFtRZWZZZCJvWLyBJ/fx44Fp4fQLbDszjJ4sruCtTTF8si0es1IsHBnMXRN74deM87nXxSnHryU7EzicUoCbxfFrscXxK/n3w/y27FNSjh3Fyz+AMVdcTZ+xF2Jn13CBY/MXHxO5/Bum3HQHg6fOtOl1aM59zrppyFKaf0YpdciyPQB4UCl1/RniXAFMryUEI5VSd9cR9hEMETjtWG3a4vDRtkz6iWh2Lf+G45HbcXBwZMCkqUTMmo+XX0CDcU1mxQ/7DG/g5JOljOzeiYenhxMR2rQ1U1uC9PwyXv/lOF9GJuFoL/xlbHdus4GXcnRGIUt2GCt+1XT8mjckkA61ZrNUShG3fze/LbV4KXcLYeyCa+k5vH4v5d9/28jqN15k8NSLmXLTnc1qu+b8pDmEYL9SakhD+2odvwD4p1JqumX7bwBKqX/XEXYfcKdSaltDtmghaBilFAmH9hO5/GsSDx/E2d2dodNnMXTG7NNGs9QXv6Y38IAgTx6a3ocLe/u2m9Eq8dnFvLLe8FL2cHbgtmbwUq7P8evqUcEMD6l7xa+aKLOZqB2/se2rzzmZlkqX3uGMW3AdwQMG/SlcRuwJlj3xMP49e3PF40/bbK1fzflFc9UIioDPMXpWFwMeSqmFZ4jjgNFZPBlIwegsXqSUOlIrXDiwBuiuGmGQFoL6MZtNHN+5jV0/fENmfAweHTsx/JJ5DJoyAyfXxrWZb7V4Ax9IyqNHZ3cemNo4b+C2yu9pBby4JooNFi/luyf1YsHIbjg7NL4vyFrHr4YwVVVxZNMGw0s5J5uQQUMZd9U1BPQKozjvJJ///T5EhMXPvtIo4dZoGkNzCIELcDtwoWXXZuAtpVRZA/FmAq9iDB/9UCn1jIg8BexWSq2whPkn4KKUerQxtmghOJ2qigqObNrA7pXfkZeRRsfAroyYcyl9x01s9Kpb+5PyeGHNMbaeyCHQy4V7p4Rx6bAgq72B2yq1vZTvnRLG/DN4KZ+N41djqaqoYP/aVez84WvDS3nkGIpO5pCVEM/Cf72gh0tqmpWzEgLLMNBPlFKLbWGctWgh+IPykmIOrPuJPat+oCQ/j4CevRk57wp6RYxu9MRw0RmFvLgmirVHM/CxeAMvOktv4LaKUorNx7N5Yc0xDqcU0MvPgwenhTG9/x9e37Udv/w6OLNgZDALbbjil+Gl/AN7Vn1PRWkps+59hPALxtvkXJrzl+aoEawBZiulKprbOGvRQmD4AOxZvZwDa1dTUVpCyKChjJx7Bd36D2x0G/4pb+Dv96Xg4eTAzRf24IZxzesN3FYxmxU/H0nnxbVRxGYVM7irF1ePDmFjVGa149e4Xr4sHm2d49fZUlKQb5nhVI+b1zQ/zSEE7wDDgBVA8an9SqmXm8vIxnI+C8HJ9FR2r/yOI5s2YK4yETZ6LCPmXo5/956NTuNYegHvbIplxYFUHOyE6y3ewG1tEY2WoMpk5ru9Kby6PprU/DK83Ry5YnjXJjl+aTRtnSY7lNUg1fJnB7Su6+Z5SEbsCXat+JbjO7Zi52BP/wmTiZh9KR0DGr9ITWR8Lm9vjGHDsUzcnOy5fkwoN4/v0aLewG0NB3s7rhzRjTlDAjmUks/AIK9zsklMozkTjRYCpdSTtjREczpKKZKOHGTX8m9IOLgPJ1c3Rswx5l13927cvDpms7Hq0lsbY9idcJJO7k7cPzWMay8Iwdvt/KsB1IeLoz0j2rBvhEZjSxotBCLSGXgY6I/hZQyAUmqSDew6rzGbTcRE7mTX8q9JjzmOu3dHxi+6nsFTL8bZrXHNFZUmMysPpPL2phiiM4oI8nblyTn9uTKiG65OusSr0Wj+wJqmoSXAl8As4DbgOiDLFkadr1RVVvL7ll+JXPEtJ9NS8PbvwtSb76LfhZMavfReSUUVX0Ym8f6WOFLySgn378ArVw1m1qDAFuvw1Gg07QtrhMBHKfWBiPxVKbUJ2CQim2xl2PlERWkJB9b/zN5VP1B0Mhe/7j2Zde+j9B51QaPmowFjnp1Ptyfw8bY4TpZUMiK0I/+a15+J4X7txhNYo9G0DtYIQaXlf5qIXILRcdzwbGWaeinJz2PvTyvZv/ZHyouLCR4wmOl33EfIwCGNzrxT80p5f0scyyITKakwMaWvH7dN6Nmm5wLSaDRtC2uE4GkR8QIeAN4APIH7bGLVOU5Bdha7ln/DkV/XUVVVSe+RFzByzuUE9AprdBrHMwp5e1Msy/enoIC5gwO5dUJPwgP0gC6NRmMd1gjBest0Evn8saKYxkoKc7NZ+viDlOTn03/CJCJmX9qoaaBPsTfxJG9tjGHd0QxcHO1YPDqEm8Z3p2vHlp17X6PRnDtYIwSHRSQD2IIxz9BWpVS+bcw6N6koK+WH//yL8pISrn725UbPI6OUYmN0Fm9tjGFXXC7ebo7cM7k3148JtemSkBqN5vzAGj+CXiISDIzHGDn0PxHJO9M01Jo/MJtNrHr9BbIS4pj/yBONEoEqk5lVh9J4a2MMx9IL6eLlwuOz+rFgRDfcz4NpIDQaTctgjR9BV2AshhAMBo4Av9nIrnOOTZ99SOyeXUy+4Xa6Dz3Nw/tPlFWa+Hp3Eu9uiSUpt5Refh68eMVg5gwOxMlBDwHVaDTNizXFykSM9QSeVUrdZiN7zkn2r1nF3tXLGTZzLkOmX1JvuPySSj7bEc9HW+PJKa5gaLA3j1/Sjyl9/dvtWgAajabtY40QDAXGAYtE5FHgOLBJKfWBTSw7R4jbt5tfPnqHnhGjmHDNDXWGSc8v48OtcSzZkUBxhYmLwjtz+4SejOzeSfsAaDQam2NNH8EBEYkBYjCahxZjLFKjhaAeshLiWPnqf+gc0p2Zdz94mnNYTFYR726K5bt9yZjMitmDA7n1wp5takF4jUZz7mNNH8FuwBnYhtE3cKFSKsFWhrV3ik7m8v1/nsLZzY15jzyOk8sfC5qcyCzkpbXR/HwkHSd7OxaMCObm8T0I9tFDQDUaTctjTdPQxUopPbdQI6gsK+OH55+irKiQBU89T4dOvtXHyqtMXPdhJAVlldx5US+uHxuKr4dzK1qr0WjOd6wRAjsR+QAIVEpdLCL9gAt0H8GfMZtNrP7vi2TGxTLv4cdPGya6ZEciKXmlfHrDSC4M69xKVmo0Gs0fWDMW8WNgDXBqJZRo4N7mNqi9s3nJx5yI3MFF191Mj2Ej/nSssKyS//56gjE9fRjf27fuBDQajaaFsUYIfJVSXwFmAKVUFWCyiVXtlAPrfmLPj98zdMZshl08+7Tj722JI7e4gkdm9NGjgTQaTZvBGiEoFhEfQAGIyGiMeYc0QPz+PWz48C26D43gomtvOu14VmE572+JZebAAAZ3824FCzUajaZurBGC+zEWru8pIluBT4G7G4okIjNEJEpETlj8D+oKc6WIHBWRIyLyhRU2tQmyE+NZ+epz+HYLYdZfH8bO/vQ1BP77y3HKq8w8OC28FSzUaDSa+mlUZ7GI2GEsTzkBCAcEiFJKVTYQzx54E5gKJAORIrJCKXW0RpjewN+AsUqpkyLi16QraSWK807y3X+exNHFlfmP/AMn19OHgCbmlPDFrkSujOhGj84erWClRqPR1E+jagRKKTPwklKqSil1RCl1uCERsDASOKGUilVKVQDLgLm1wtwMvKmUOmk5V6YV9rcqleXGMNHSwgLmP/wEHXzq7gB+aV0UdiLcO6V3C1uo0Wg0DWNN09BaEblMrOvlDAKSamwnW/bVJAwIE5GtIrJDRGbUlZCI3CIiu0Vkd1ZW67szKLOZn958mfTYE1xy90P49+hVZ7gjqfks35/KX8Z2x9/TpYWt1Gg0moaxxo/gfsAdqBKRMozmIaWUOtN8CHWJhqrDht7ARRhLX24RkQFKqbw/RVLqXeBdgIiIiNpptDhbln7C8Z3buOjam+g1YnS94V5YE4WniwO3T+jZgtZpNBpN42l0jUAp1UEpZaeUclJKeVq2q0VARPrXES0Z6FZjuyvGWse1wyxXSlUqpeKAKAxhaLMc3PAzkSu+ZfDUmQybWbul6w92xOawMSqLOyb2wsvNsQUt1Gg0msbTnJPbf1bHvkigt4h0FxEnYAHGyKOa/IBl6UsR8cVoKoptRrualYSD+1n//v8IHTKcSX+5tV5/AKUUz/10jABPF64fE9qyRmo0Go0VNKcQnJYjWpzO7sLwSP4d+EopdUREnhKROZZga4AcETkK/Ao8pJTKaUa7mo2c5ERWvvJvfLoGM+uvj9Q5TPQUa45ksD8pj3un9MbFsf5wGo1G09o053qHdbbbK6VWA6tr7Xuixm+F0f9wfzPa0uwU553ku+eexMHJifmPPIGzW/0zhVaZzLyw5hg9O7tz+fDGL0yv0Wg0rYFe97ARVFaUs/yFpynJz2Pew0/g6XtmV4dv9yYTk1XMQ9PDcbDXt1ij0bRtmjOXqmjGtNoMymzm5zdfIS0mmpl3P0BAzzP3Y5dVmnhl3XEGd/Nmev+AFrJSo9Fomo5VTUMiEgSE1IynlNps+V//GMp2zG9ffkb0jt+4cPEN9B45psHwn26PJ72gjJevGqwnltNoNO0Ca1Yo+w9wFXCUP2YdVcBmG9jVJjj86zp2/fA1gybPIGLW/AbD55dW8uavMVwY1pkxPfU00xqNpn1gTY1gHhCulCq3lTFticTDB1j33n8JGTSUSTfc1qjS/TubYsgvreTh6XpiOY1G036wpo8gFjgvvKJyUpJY8fKzdOwSxOz7HsXeoWG9zCgo48OtccwZHMiAIK8WsFKj0WiaB2tqBCXAfhHZAFTXCpRS9zS7Va1ISUE+3z/3T+wdHJn/yD9wdnNvVLzXNhynyqR4YFqYjS3UaDSa5sUaIVjB6V7B5xRVFRUsf+Fpik+e5Mp//BsvP/9GxYvNKuLLyCSuHhVMiE/jhEOj0WjaCo0WAqXUJ5ZpIk4VeRtcj6A9ocxmfn7rVVKjf2f2fY/SpXfj2/lfWhuNs4Mdd09q01MkaTQaTZ1YM2roIuATIB5jOoluInLdqeGj7Z1tXy8hattmxi+6nrDR4xod72ByHqsOpXHPpF507uBsQws1Go3GNljTNPQSME0pFQUgImHAUmC4LQxrSY5s2sCO775kwMRpjJhzmVVxn/85io5ujtx8YQ8bWafRaDS2xZpRQ46nRABAKRXNOTCKKOnIQda+8wbBAwYz5aY7rHIC++14Nr+dyObOib3o4NLub4VGozlPsaZGsFtEPuCP6aavBvY0v0ktR25qMiteehbvgC7Mvv9vjRomegqzWfGfn48R5O3K4tEhNrRSo9FobIs1NYLbgSPAPcBfMTyMb7OFUS2BMUz0ScTenksf/Qcu7tYtKr/6cBqHUvK5b2qYnmZao9G0a6wZNVQOvGz5a9dUVVay4qVnKMzN5sonnsXLz7rJ4SpNZl5cE0W4fwfmD629BHP7QSml50PSaDQN1whE5CvL/0MicrD2n+1NbF6UUqx561VSjh3l4jvvJzCsr9VpfBmZRHxOCQ9ND8fern1mpBWlJax6/QUOb1zf2qZoNJpWpjE1gr9a/s+ypSEtxfZvvuDY1k2MW3At4ReMtzp+SUUVr204TkRIRyb3PfO6BG2VrMR4Vr78b/LS0xqcVluj0Zz7NCgESqk0y887lFKP1DxmmZH0kdNjtU2ObvmV7d8spf+EKYycd0WT0vhoazxZheX87+ph7bJZ5fDG9Wz44C2c3d254oln6NZvYGubpNFoWhlrOoun1rHv4uYypCWoqignZNBQpt5yZ5My8bySCt7eFMPkPn6MCO1kAwttR2V5GT+/9Spr3nqVwLBwrnnuNS0CGo0GaESNQERuB+4AetTqE+gAbLWVYbZg0OQZDJw4DbFr2sJs/9sYQ1F5FQ/NaF/TTOemJrPylefITkpg9GULuODyhdjZ6ZFOGo3GoDF9BF8APwH/Bh6tsb9QKZVrE6tsSFNFIDWvlI+3xTN/aBB9Ajyb2SrbEbV9C2vefh0HR0cue/SfhA5p947gGo2mmWlMH0E+kA8sBBARP8AF8BARD6VUom1NbBu8uj4aFNw/tX1MM11VWcmmzz5g/5ofCQzry6x7H6GDj141TaPRnI41k87NxvAhCAQyMdYu/h3obxvT2g7HMwr5Zk8y14/pTteObq1tToPkZ2bw46vPkR5znOGz5jN+4XVWeU1rNJrzC2vaSZ4GRgPRSqnuwGQa0UcgIjNEJEpETojIo3Ucv15EskRkv+XvJitsahFeWBOFm5MDd03q1dqmNEjMnp189ug9nExLZc6D/8dF19yoRUCj0ZwRa3KISqVUjojYiYidUupXy/DRehERe+BNjBFHyUCkiKxQSh2tFfRLpdRd1pneMuxNPMnaoxncPzWMTu5OrW1OvZiqqtj65WdErvgWv+49mX3f3/D2t85jWqPRnJ9YIwR5IuIBbAaWiEgmUNVAnJHACaVULICILAPmYsxT1OZRSvGfn47h6+HEjeO6t7Y59VKYm82q154n5dhRBk+dyUXX3oSDU9sVLY1G07awRgjmAqXAfRgzj3oBTzUQJwhIqrGdDIyqI9xlInIhEA3cp5RKqh1ARG4BbgEIDg62wuymszE6i51xuTw5pz/uzm2zeSX+4D5Wv/EiVeXlzLz7QfqOu8iq+MqskHY6TYZGo2kerMnd/IA0pVQZ8ImIuAL+QM4Z4tSVw6ha2yuBpUqpchG5DWMVtEmnRVLqXeBdgIiIiNppNDtms+L5n6MI7uTGygKZSgAAFodJREFUwpEtIzzWYDab2PHtMrZ/uwyfoG7Mvv9v+AR1a1RcVWmi5EA2RTvTcO3TCc/Jbe/6NBpNy2GNEHwNjKmxbbLsG3GGOMlAzdypK5BaM4BSqqaQvAecsd+hpVhxIJXf0wp4bcEQnBya5ntgK0ry81j1xoskHtpPvwsnMeXGO3B0cWkwXmVWCcU70ijek4kqq8Khsyv2HfXymhrN+Y41QuCglKo4taGUqrAsZn8mIoHeItIdSAEWAItqBhCRLjXmM5qDMSS1VamoMvPSuij6dfH8//buPL6q+szj+OfJBllYImtYBURcAFEj2GIrYgXci1VHW1u1tkxnXGAcBZ1aFZepdSnaqTNKoVXHrdXiCJZWq1LEWpSAVlY1gNZAAgiBkISQ5T7zx73ATYiQqzm5Ief7/if3rHnueSXnuff8fs/vx7nDeyU7nHqKVq/gDw/dS1V5OeN+dB1Dx5xxwOEyvC7CrlVbqVhczO61OyDFyBzahexRebQb2OmQHC9JRJpXIolgi5md5+5zAczsfOCzAx3g7rVmdg3wMpAK/NrdV5rZHUBB7FzXmdl5RBuetwFXfIH30ayefvsTPt22i8euHEpKK3l+7pEIS+bN4c1nn6Bzj55MvOl2uh/++fMk127fTcU7xVQsKSGys4bUzu3oOL4/2fk9Se2ghmQR2cfcm/a43cwGAU8RLSgzoo3A33P3wuDCa1x+fr4XFBQEcu7y3bWceu8CBvfI4ZkfntwqPjHvKt/Jn/57BuuWvsORo0Yz7keTaZe1f2GbR5yqj0qpWFxM1Zro6B/thxxG9qietB9ymBqFRULOzJa6e37D9YnMULYWODnWhdTcfWdzBthazF60nq0V1cyacFSrSAIlhR8y78F7KN+2jdOu+GeOn3DOfnHVlVdTUbCJindKqNtWRUpOOh1O7Uv2yJ6kHXbwtgMRCbemjD56mbs/aWbXN1gPgLsf8lNX7rG1fDcz31jL+GN7cHy/3KTG4u689/JL/OWJ2WTn5nLJ9J+RN3hIve3VH5dRvriYXSs+gzonY0AnOo0/nMxju2CtrIFbRFqvpnwj2PMMokOQgbQGv1xQyK6aOm4cn9xhpndXVvLKzP/iw78tYuAJJzHh6uvJzIle/khVLZXLNlP+djG1myqx9qnkjMoje1RP0ntkJzVuETk0NSURDIr9XOXuzwUZTDJ9uq2Spxb/g4tO7MsR3ZOX87Z8sp55M37K9k0lfO3bV3DSuRdgKSlUbyin4u1iKt/bjFdHSO+dQ+63BpN5XDdSMjS3gIh8cU1JBGeZ2S3AzUTrBtqkGX/+EDOYckby5vBdvuAVXp/9CO1ycrj4J/9J78FHU7lsCxVvF1P96U4sPYXM47qRMyqPjL5t/guaiLSQpiSCPxHtJpptZmVx6w1wdz90Zmn5HKuLy3jhvQ1M+tpA8jpltvjvr9ldxWuzH2HlwlfpN/Q4xn/7WiKrK9n43Dv4rmjhV6dzBpJ9QndSstJbPD4RaduaMjHNjcCNZvaiu5/fAjG1uPtf/oAO7dL4lzGDDr5zM9u64VNemnEPW4uKGDv2+/S2Iyj71Ucq/BKRFpNI99E2mQSWfLyN19ZsZuqEIXTOatlCqzV/XciiWY8xsMNwxh59KbbeqetcRcdx/ck+SYVfItIymtJ99E13P8XMdhIdMC7+o+kh/WjI3bnnj2vo3qEdV3615YaZrtldTcEjz5JSWMuEHldiGO37dSb75DwVfolIi2vKo6FTYj/bXOvkq6s3s/STUu6eOJTMFup5U/ZBMRt/U0BfBlDbsZac0X3ocHIvFX6JSNIkMmfxIKAoNlz0GGA48IS7bw8quCDVRZz7Xl7DgK7ZXJzftOGbv4xIdR1b/m8l1UtLSY2ksXukMXDiqSr8EpGkS+Qu9HugzsyOAGYDA4CnA4mqBcxZVsSHm8q5YdwQ0lODvRlXFZay8b7F1CzbwSdVq8n6Xn8GXXSKkoCItAqJjD4aiY0mOhF40N3/y8zeDSqwIFXV1DHjzx8yvE8nzhoW3Ly+kcoatv9hPZVLN7GzppTVkXc47eZ/Jbdn6xjaelftLmoiNXTMOGSbeUSkGSQ0eb2ZXQpcDpwbW3dIdmp/cvEnbNxRxX0XHRdIt0x3Z9eKz9j+4lrqyqtZvX0x27ps4bxpt5DVsVOz/77GRDzC1l1bKa4opriimJKKkujr8n3LpbtL+c7R3+GmkTe1SEwi0jolkgiuBH4E3O3u62OTzTwZTFjBKauq4eEFhXxtcFdGH9G12c9ft2M3pS+upWrVVqraV7FwwzN0PW4Q37r2TtLbNV+DcGVN5d6b/J4bfElFCSWVJdHXlSXURmrrHZOVlkWvnF70zO7J0K5DycvOY0T3Ec0Wk4gcmhKpI1gFXAdgZrlAB3e/J6jAgvKrN9ZRWlnD1PFHNet5PeJULClhx/z1eMQpylnHW8ufZ8SZ5zDmez8gJaXpvZLqInVs2bVl36f4uBv9nuWy6rJ6x6RaKt2zupOXncfwbsMZlz2OvOw88rLz6Jndk7ycPDqkd1BhmojsJ5FeQ38hOpVkGvAe0RnLFrr79Qc8sBXZvLOKWYvWc/bwPIb1ab5HNDVbKimdU0j1+h2k9c/hbxv+j3UrljHme1dx4tnf3H//uhrWl62P3thjn97jb/abKjdR53X1jumY0XHvTX1E9xF7b/J5OdGfXTO7kpaSyBc8EZGoRO4cndy9zMx+APzG3W8zs/eDCiwIv3y9kJq6CDeMa55hpr0uws43NlD22idYWirtzujOvHkPUPbZFs6dMo0jTz5lv2PW71jPda9fx8dlH+9dl5aSRo+sHuRl53FijxP3foKP/0Sfna4hpkUkGAlNXm9mecDFwI8DiidQV50ygKG9OzGg65e/qVYX7aT09x9RU1xB5tAu7B4Gz/3yDjwS4aJb7qb3Ucfsd8yiokVMfWMqGakZ3DX6LgZ0GkBedh5dMruQYupKKiLJkUgiuIPoJPRvuvsSMxsIfBRMWMHo3yWb/l2+XBKIVNdR9uonlC/aQEpOBl0uO5oNuwr5w/33kZ2bywU3TeewXr3rHePuPLbyMWYsncGQw4bwi9N+QV5O3peKQ0SkuSTSWPwccfMRuPs64FtBBNVaVRWWUjqnkLptVWSP7EmnMwfw94V/5PXHZtJz4BFMnHYbWZ061z+mtorpf5vOS+teYlz/cdw5+k6y0vefeF5EJFkSaSxuD1wFHAvs7Qfp7t8PIK5WJVJZw/b566ks2ERa10y6TRpGxuEdeePpxyiYN4dB+aM4+9obSW9fv3vopopNTFkwhRVbV3Dt8dfyw2E/VK8dEWl1Enk09L/AGmA80cdE3wFWH+wgM5sAPASkArM+r8upmV1I9BvHSe5ekEBcgYkvDItU1tBhTB86nt6POq/jpV/cx4d/W8SI8Wdz2hWT9use+v6W95myYAoVNRU8dNpDjO03NknvQkTkwBJJBEe4+0Vmdr67P25mTxNtM/hcZpYKPAycARQBS8xsbqwmIX6/DkRrFN5OLPzgxBeGpffOoeuVQ8noncOu8p28eN+dbFiziq9f9n3yz5m436f8uWvnMv2t6XTL6sajZzzK4NzkTX8pInIwCQ0xEfu53cyGAiXA4Qc5ZiRQGGtPwMyeBc4HVjXY707gXuCGBOIJRHxhGBGn01kDyBndG0s1dmwu4fc/vZ2yzSWcPXkqR3316/WOrYvUMWPpDB5f9Tgje47kgVMfoHP7zp/zm0REWodEEsHMWEXxT4C5QA5w60GO6Q18GrdcBIyK38HMjgf6uvtLZpbURBBfGNZuUCdyLxhMWpfoHMYlaz/ihZ9NJ1Jby4W33EWfo4fWO7asuoypb0zlrxv+yqVHXcqNJ91IesohORSTiIRMIr2GZsVeLgQGNvGwxlpGfe9GsxRgBnDFQU9kNgmYBNCvX78m/vqm8boIOxdtoOzVaGFY7rcGk5XfY+8jn7VL3+Glh35GVsfOXHDrT+nSp/78BXuKxIrKi7jtK7dx4ZEXNmt8IiJBaspUlQccQsLdf36AzUVA/F2zD7AxbrkDMBT4S+ym2xOYa2bnNWwwdveZwEyA/Px8p5k0LAzrfN4RpHbcN1fw3/88n9dmP0L3AQOZOO02sjvn1jt+UdEipr0xjfTUdGaPm80JPU5ortBERFpEU74R7JmisuF8xXvWHcgSYHBspNINwCXAt/ce7L4D2DsEaGw8oxtaotdQY4VhmUP3jUbqkQiLnn2CJS8+z8ATTuLsyVPJaJ+5b7s7j698nBnLZnBk7pEqEhORQ1ZT5iyeDmBmjwOT90xNGWsveOAgx9aa2TVEexelAr9295VmdgdQ4O5zv+wb+CIaKwxLydx3KWpranj5fx5kzV8XMvwbEzj9+/9CSuq+7qG763Zz+1u3q0hMRNqERBqLh8fPT+zupbGG3gNy9/nA/AbrGm1kdvcxCcSTsIaFYV1/OIz2gxpUApeX8+IDd1G0agWnXHo5I8+/sF730M2Vm5myYArLP1vONSOuYdLwSSoSE5FDWiKJIMXMct29FMDMDkvw+KTbPn89lcs27S0Ms/T6RWBlWzYz557bKS3eyFnX3sDRp4yptz2+SOzB0x7k9H6nt2D0IiLBSORG/gDwlpk9T7Rt4GLg7kCiCkincf3J+UovMnrn7Ldt07pCXvjZdGqrq7nwx3fQ99jh9barSExE2qpEuo8+YWYFwFiijcYXNKwQbu1SO7YjtWO7/davf7eAeTPuoX2HDlxyy1107dt/77a6SB0PLnuQx1Y+piIxEWmTEnq0E7vxH1I3/4N5/7WXeXXWw3TrN4CJ024l57Aue7epSExEwuCQesbfnNydt373JIvn/JbDR5zIuVOmkZG5r+fP3iKxnUXc+pVbuejIi5IYrYhIcEKZCOpqa3jlkV+watECho0dx+lX/SupafsuRXyR2Kzxszixx4lJjFZEJFihSwS7KyuY+8Dd/GPF+4z+p+8yauLFe7t/NiwSe+i0h+iV0yvJEYuIBCtUiaDssy28cM/tbNtYxJlXX88xX983R4CKxEQkrEKVCN6e81vKPtvCBTdPp/+wEXvXq0hMRMIsVIlgzOU/4ISzzq83eujyLcuZvGAy5TXlKhITkVBKSXYALSm9Xft6SWDe2nlc8acryEjN4MmznlQSEJFQCtU3gj0aFondf+r95LbPPfiBIiJtUOgSQXyR2CVDLmHqyKkqEhORUAtVIlCRmIjI/kKVCJ5Z8wxl1WUqEhMRiWPuzTbrY4vJz8/3goLEJzGrrqtmW9U2emb3DCAqEZHWzcyWunt+w/Wh6jWUkZqhJCAi0kCoEoGIiOxPiUBEJOSUCEREQk6JQEQk5JQIRERCTolARCTkAk8EZjbBzD4ws0Izu6mR7T8ys+Vm9p6ZvWlmxwQdk4iI7BNoIjCzVOBh4EzgGODSRm70T7v7MHcfAdwL/DzImEREpL6gvxGMBArdfZ27VwPPAufH7+DuZXGL2cChV+osInIIC3qsod7Ap3HLRcCohjuZ2dXA9UAGMLbh9tg+k4BJAP369Wv2QEVEwirobwSNzfe43yd+d3/Y3QcB04BbGjuRu89093x3z+/WrVszhykiEl5BJ4IioG/cch9g4wH2fxb4ZqARiYhIPUEngiXAYDMbYGYZwCXA3PgdzGxw3OLZwEcBxyQiInECbSNw91ozuwZ4GUgFfu3uK83sDqDA3ecC15jZN4AaoBS4PMiYRESkvsAnpnH3+cD8ButujXs9OegYRETk86myWEQk5JQIRERCTolARCTklAhEREJOiUBEJOSUCEREQk6JQEQk5JQIRERCTolARCTklAhEREJOiUBEJOSUCEREQk6JQEQk5JQIRERCTolARCTklAhEREJOiUBEJOSUCEREQk6JQEQk5JQIRERCTolARCTklAhEREJOiUBEJOQCTwRmNsHMPjCzQjO7qZHt15vZKjN738xeM7P+QcckIiL7BJoIzCwVeBg4EzgGuNTMjmmw27tAvrsPB54H7g0yJhERqS/obwQjgUJ3X+fu1cCzwPnxO7j7AnevjC0uBvoEHJOIiMRJC/j8vYFP45aLgFEH2P8q4I+NbTCzScCk2GK5mX3wBWPqCnz2BY9ti3Q99tG1qE/Xo762cD0affQedCKwRtZ5ozuaXQbkA6c2tt3dZwIzv3RAZgXunv9lz9NW6Hrso2tRn65HfW35egSdCIqAvnHLfYCNDXcys28APwZOdffdAcckIiJxgm4jWAIMNrMBZpYBXALMjd/BzI4HHgXOc/fNAccjIiINBJoI3L0WuAZ4GVgN/M7dV5rZHWZ2Xmy3+4Ac4Dkze8/M5n7O6ZrLl3681Mboeuyja1Gfrkd9bfZ6mHujj+xFRCQkVFksIhJySgQiIiEXqkRwsOEuwsLM+prZAjNbbWYrzWxysmNqDcws1czeNbOXkh1LsplZZzN73szWxP5OvpLsmJLFzP4t9n+ywsyeMbP2yY6puYUmETRxuIuwqAX+3d2PBk4Grg7xtYg3mWinBoGHgD+5+1HAcYT0uphZb+A6osPgDAVSifZ+bFNCkwhownAXYeHuxe6+LPZ6J9F/8t7JjSq5zKwPcDYwK9mxJJuZdQS+DswGcPdqd9+e3KiSKg3INLM0IItGaqEOdWFKBI0NdxHqmx+AmR0OHA+8ndxIku5BYCoQSXYgrcBAYAvwm9ijsllmlp3soJLB3TcA9wP/AIqBHe7+SnKjan5hSgRNHu4iLMwsB/g9MMXdy5IdT7KY2TnAZndfmuxYWok04ATgf9z9eKACCGWbmpnlEn1yMADoBWTHhsNpU8KUCJo03EVYmFk60STwlLvPSXY8STYaOM/MPib6yHCsmT2Z3JCSqggocvc93xKfJ5oYwugbwHp33+LuNcAc4KtJjqnZhSkRHHS4i7AwMyP6/He1u/882fEkm7vf7O593P1won8Xr7t7m/vU11TuXgJ8amZDYqtOB1YlMaRk+gdwspllxf5vTqcNNpwHPehcq+HutWa2Z7iLVODX7r4yyWEly2jgu8ByM3svtu4/3H1+EmOS1uVa4KnYh6Z1wJVJjicp3P1tM3seWEa0t927tMGhJjTEhIhIyIXp0ZCIiDRCiUBEJOSUCEREQk6JQEQk5JQIRERCTolApIWZ2RiNcCqtiRKBiEjIKRGIfA4zu8zM3onNpf1obL6CcjN7wMyWmdlrZtYttu8IM1tsZu+b2QuxMWowsyPM7FUz+3vsmEGx0+fEjff/VKxqVSQplAhEGmFmRwP/BIx29xFAHfAdIBtY5u4nAAuB22KHPAFMc/fhwPK49U8BD7v7cUTHqCmOrT8emEJ0boyBRKu9RZIiNENMiCTodOBEYEnsw3omsJnoMNW/je3zJDDHzDoBnd19YWz948BzZtYB6O3uLwC4exVA7HzvuHtRbPk94HDgzeDflsj+lAhEGmfA4+5+c72VZj9psN+Bxmg50OOe3XGv69D/oiSRHg2JNO414EIz6w5gZoeZWX+i/zMXxvb5NvCmu+8ASs3sa7H13wUWxuZ4KDKzb8bO0c7Mslr0XYg0gT6FiDTC3VeZ2S3AK2aWAtQAVxOdpOVYM1sK7CDajgBwOfBI7EYfP1rnd4FHzeyO2DkuasG3IdIkGn1UJAFmVu7uOcmOQ6Q56dGQiEjI6RuBiEjI6RuBiEjIKRGIiIScEoGISMgpEYiIhJwSgYhIyP0/CzAU8uXM91sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on Validation Data: 0.825\n"
     ]
    }
   ],
   "source": [
    "num_train_epochs = 10 # How many epochs to train each network. Small value used just for quick demo, you should increase it.\n",
    "num_hpo_trials = 16 # How many networks to train with different hyperparameters. Small value used just for quick demo, you should increase it.\n",
    "ngpus_per_trial = 1 # How many GPUs to use per training run. Set =0 if you don't have GPU but beware of slow run-times (otherwise make sure GPU-version of MXNet is installed).\n",
    "\n",
    "classifier = task.fit(dataset, plot_results=True,\n",
    "                      search_strategy='hyperband', scheduler_options={'grace_period':1},\n",
    "                      net=net, optimizer=optimizer, lr_scheduler=ag.space.Categorical('poly', 'cosine'),\n",
    "                      epochs=num_train_epochs, num_trials=num_hpo_trials, ngpus_per_trial=ngpus_per_trial)\n",
    "\n",
    "print('Accuracy on Validation Data: %.3f' % classifier.results['best_reward'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that AutoGluon early-terminated many training jobs whose performance in early epochs was significantly worse than in the same epochs of previously-completed training runs. [Hyperband](https://homes.cs.washington.edu/~jamieson/hyperband.html) is based on successive halving of randomly-sampled configurations, where we terminate the worst-performing 50% of the current training jobs every few epochs. However  this may be too aggressive; some good hyperparameter-configurations may just happen to exhibit poor performance at a certain training stage but would end up producing good performance at the end of training. To ensure such configurations have a chance of surviving, Hyperband runs successive-halving in numerous brackets where random networks in every bracket are trained for all epochs. Below we show what training-performance might look like over the course of 4 different Hyperband brackets (one per plot):   \n",
    "\n",
    "<img src=\"files/images/hyperband.png\" width=\"500\" height=\"400\">\n",
    "\n",
    "AutoGluon actually uses an [asychronous version of Hyperband](https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/) that is slightly more complex. \n",
    "Finally, we evaluate the resulting model from our Hyperband-search on the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec6d26dbdd5c457896abaef9b78a9151",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, max=1), HTML(value=''))), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy on test data: 0.891\n"
     ]
    }
   ],
   "source": [
    "test_acc = classifier.evaluate(test_dataset)\n",
    "print('Accuracy on test data: %.3f' % test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can try playing with AutoGluon's various [searchers](https://autogluon.mxnet.io/api/autogluon.searcher.html) & [schedulers](https://autogluon.mxnet.io/api/autogluon.scheduler.html) to see how they affect the resulting image-classification accuracy (make sure to increase `num_train_epochs`, `num_hpo_trials` to get a more realistic picture). \n",
    "For large-scale HPO, you can also use AutoGluon to easily distribute these HPO jobs over multiple machines, as demonstrated in this [Distributed Search tutorial](https://autogluon.mxnet.io/tutorials/course/distributed.html). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[**AutoGluon Documentation** (autogluon.mxnet.io)](https://autogluon.mxnet.io/api/autogluon.task.html)\n",
    "\n",
    "Feurer M, Hutter F. [**Hyperparameter Optimization**](https://www.automl.org/wp-content/uploads/2019/05/AutoML_Book_Chapter1.pdf). Book chapter in *AUTOML: METHODS, SYSTEMS, CHALLENGES*, 2018.\n",
    "\n",
    "Agnihotri et al. [**Exploring Bayesian Optimization**](https://distill.pub/2020/bayesian-optimization/). *Distill*, \n",
    "2020.\n",
    "\n",
    "Shahriari et al. [**Taking the Human Out of the Loop: A Review of Bayesian Optimization**](https://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf). *Proceedings of the IEEE*, 2015.\n",
    "\n",
    "Jamieson et al. [**Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization**](https://homes.cs.washington.edu/~jamieson/hyperband.html). *Journal of Machine Learning Research*, 2018.\n",
    "\n",
    "Li et al. [**Massively Parallel Hyperparameter Optimization**](https://arxiv.org/abs/1810.05934). In: *MLSys*, 2020."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "0982f62e3b804c17b5d7e00256424583": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "100874d357e042c5b51867c83704b727": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "1177ea817a95431995cd4d75c02e3ded": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "121741fb798246d4b526327ea488b022": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "1505da151ead4085bbbf705d10848daa": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "17d1be76ede841df85e10e06d95e3c22": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_2b98d70a7bf044a4bca57c19b2ef984a",
       "style": "IPY_MODEL_1177ea817a95431995cd4d75c02e3ded"
      }
     },
     "1f51d77c1e3646e7993d91b84c2c537b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "VBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_8c302fb0abdf4e51b32ddfb37d5502c7",
        "IPY_MODEL_17d1be76ede841df85e10e06d95e3c22"
       ],
       "layout": "IPY_MODEL_d0b05d7f7bfa4e5fb6013a8f9b2a68a2"
      }
     },
     "1f88b426c78b480a8b5d44ce107791a2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "accuracy: 0.9375: 100%",
       "layout": "IPY_MODEL_f586be9e836348fb890cac4e388a9e17",
       "max": 1,
       "style": "IPY_MODEL_78150a836be94270a384aab49371b04b",
       "value": 1
      }
     },
     "252273961265411792197d38c6f97407": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "27d129eb01de4de697dd78c0bb55175a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "accuracy: 0.953125: 100%",
       "layout": "IPY_MODEL_efe543f8e88f4022a2d232ba7a02d5d1",
       "max": 1,
       "style": "IPY_MODEL_bdf2570d34e6431ca29844331fa2b94f",
       "value": 1
      }
     },
     "2856731bc3eb4611a7f26878a7692a8b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_e655a826b7fb4c08bf5c8a6c7716488e",
       "max": 16,
       "style": "IPY_MODEL_5ee1568d135643b3b0e9274ef13b33d1",
       "value": 16
      }
     },
     "2ae08a2cbeda4e20a191d96103824293": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "2b98d70a7bf044a4bca57c19b2ef984a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "2eae6a1ee3a645e0b343c049110ba4f3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "31b2a6e102dc4140bd96f438334a0ea1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_27d129eb01de4de697dd78c0bb55175a",
        "IPY_MODEL_6b04a810e9d34faaa46a1d5bf5875d8a"
       ],
       "layout": "IPY_MODEL_b98ae88fb2ed4959820899e4602b43f6"
      }
     },
     "3358be381a7549a5a6a30f2909631f44": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "36f44534cbc0438da077a6938d1b2ced": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_0982f62e3b804c17b5d7e00256424583",
       "max": 2,
       "style": "IPY_MODEL_caf73e848e8c459b8e4b0ad2b4d0e345",
       "value": 2
      }
     },
     "3b9732bb27b14e128c9312312f1f4a48": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "42f1cac1905145b086b99aec1500ced9": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "4394cb10c88b470189a17ee1849d2804": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_92dde42283104ba8875feb49b65405f5",
       "style": "IPY_MODEL_cc91584c6d7e4d75ae156344b81fbf50",
       "value": " 1/1 [00:06&lt;00:00,  6.30s/it]"
      }
     },
     "48146fd39af240d1accfb8ca595b20a8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "48f75ecc31f8462ca4e711b8775ef446": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "4b1e1dd73e9d49e9890c8523fde6b7b3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_60b38a806aeb49a4968bfc62853ad0b5",
       "style": "IPY_MODEL_48146fd39af240d1accfb8ca595b20a8",
       "value": " 2/2 [03:08&lt;00:00, 94.13s/it]"
      }
     },
     "4e4a7f9625a04f40802819b5464ffd62": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_a07651634c8941189cfd35902258c8a6",
       "style": "IPY_MODEL_eb194c6fd8534dbd808a17ab4a3d24a5"
      }
     },
     "57fc909a20ab427f820a980e679bf6f0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_69ac9495e6c54d26b09fa41a2882d9de",
       "style": "IPY_MODEL_9f5e38a1f03d49bebe9cc22cd6142667"
      }
     },
     "5ee1568d135643b3b0e9274ef13b33d1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "60b38a806aeb49a4968bfc62853ad0b5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "63321d057e774122ae6ae5e99f55a0f9": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "633df7fdc3ed41869f7c1edc9e4004cb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_63321d057e774122ae6ae5e99f55a0f9",
       "style": "IPY_MODEL_100874d357e042c5b51867c83704b727",
       "value": " 2/2 [00:12&lt;00:00,  6.28s/it]"
      }
     },
     "69ac9495e6c54d26b09fa41a2882d9de": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6b04a810e9d34faaa46a1d5bf5875d8a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_8ee69313fabb4c5692825e9f45b76341",
       "style": "IPY_MODEL_99c95d598fb541fe98e7db9967552596",
       "value": " 1/1 [09:55&lt;00:00, 595.93s/it]"
      }
     },
     "6fcf8d85db8c46e989c1a2e55575edde": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "763e0e4ea0b048be998b439386c54cec": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_2856731bc3eb4611a7f26878a7692a8b",
        "IPY_MODEL_d6bfff7369214641aa1675f87cf5d582"
       ],
       "layout": "IPY_MODEL_9eb22eae95bd4a6aaecbf93e8502cae7"
      }
     },
     "78150a836be94270a384aab49371b04b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "78e775f4f6ae45a08ae23f4d26221362": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "7b12843b2bf742809f03e2e3d641453f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_af72e6ad8c0a49f2912435f569531547",
       "style": "IPY_MODEL_96242e23a7544492b0d2e9ed9099b6e6",
       "value": " 1/1 [00:05&lt;00:00,  5.99s/it]"
      }
     },
     "7b831536a9f0412f9f83a9588307677b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_ddc3b2718447459db97facd28200544f",
        "IPY_MODEL_abadb2dce67f40daa63b275841b179af"
       ],
       "layout": "IPY_MODEL_b267da25ab5841ddb5fb49535a967bc5"
      }
     },
     "856e0780a1bb45e0a9355947f6084824": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "878d9ba4eec24048a78546f7c81741c9": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "8791d2693b7a4fd78d05095dc5c58582": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "8c302fb0abdf4e51b32ddfb37d5502c7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_1f88b426c78b480a8b5d44ce107791a2",
        "IPY_MODEL_4394cb10c88b470189a17ee1849d2804"
       ],
       "layout": "IPY_MODEL_1505da151ead4085bbbf705d10848daa"
      }
     },
     "8ee69313fabb4c5692825e9f45b76341": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "8fd95eecc3034dc982a64c1a379405ca": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "92dde42283104ba8875feb49b65405f5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "96242e23a7544492b0d2e9ed9099b6e6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "99c95d598fb541fe98e7db9967552596": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "9eb22eae95bd4a6aaecbf93e8502cae7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "9f5e38a1f03d49bebe9cc22cd6142667": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "a07651634c8941189cfd35902258c8a6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "a2d43573f6644bccae17f93d9f2ca25e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_878d9ba4eec24048a78546f7c81741c9",
       "style": "IPY_MODEL_2eae6a1ee3a645e0b343c049110ba4f3"
      }
     },
     "a2eccfb447224ca0bb310fb6b19ba85a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_ec5088958a4d464389421e0bc66f286b",
        "IPY_MODEL_7b12843b2bf742809f03e2e3d641453f"
       ],
       "layout": "IPY_MODEL_da03a67e2f7d4365a5d8460a1fe7bac3"
      }
     },
     "abadb2dce67f40daa63b275841b179af": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_fb306bd8776f459b8f32af1e73a0405a",
       "style": "IPY_MODEL_48f75ecc31f8462ca4e711b8775ef446",
       "value": " 16/16 [00:43&lt;00:00,  2.73s/it]"
      }
     },
     "ac4a02c7fd39421f8d64a1f8f3f60737": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "af72e6ad8c0a49f2912435f569531547": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "b267da25ab5841ddb5fb49535a967bc5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "b5ca196fe0a14af99966fc60ca78fc6b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "accuracy: 0.890625: 100%",
       "layout": "IPY_MODEL_b63798a78cd948a5ab085ded25d747a3",
       "max": 1,
       "style": "IPY_MODEL_6fcf8d85db8c46e989c1a2e55575edde",
       "value": 1
      }
     },
     "b63798a78cd948a5ab085ded25d747a3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "b8bf577b1e33455cab4e48e7aa6bd18d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_d80e318c5197499f8a65f4dc8ae88477",
        "IPY_MODEL_4b1e1dd73e9d49e9890c8523fde6b7b3"
       ],
       "layout": "IPY_MODEL_3358be381a7549a5a6a30f2909631f44"
      }
     },
     "b98ae88fb2ed4959820899e4602b43f6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "bdf2570d34e6431ca29844331fa2b94f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "c5f998bd592848d6a34210e43242c2d2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_3b9732bb27b14e128c9312312f1f4a48",
       "style": "IPY_MODEL_8791d2693b7a4fd78d05095dc5c58582",
       "value": " 1/1 [00:06&lt;00:00,  6.52s/it]"
      }
     },
     "c620f07561c2403ea6ab74be440e9697": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "c778782ad4de4ddb88c2f921b21f9d30": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "caf73e848e8c459b8e4b0ad2b4d0e345": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "cc91584c6d7e4d75ae156344b81fbf50": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "cd2851c6d417407bbe69c19a2cf14c74": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "VBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_a2eccfb447224ca0bb310fb6b19ba85a",
        "IPY_MODEL_a2d43573f6644bccae17f93d9f2ca25e"
       ],
       "layout": "IPY_MODEL_ac4a02c7fd39421f8d64a1f8f3f60737"
      }
     },
     "d0b05d7f7bfa4e5fb6013a8f9b2a68a2": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "d4bc994409ef49b98ca012846f28dd78": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_36f44534cbc0438da077a6938d1b2ced",
        "IPY_MODEL_633df7fdc3ed41869f7c1edc9e4004cb"
       ],
       "layout": "IPY_MODEL_252273961265411792197d38c6f97407"
      }
     },
     "d6bfff7369214641aa1675f87cf5d582": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_ecf0026e5155445991c02373a5681b55",
       "style": "IPY_MODEL_d7a16264121f462ab95155910e692bc9",
       "value": " 16/16 [09:49&lt;00:00, 36.87s/it]"
      }
     },
     "d7a16264121f462ab95155910e692bc9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "d80e318c5197499f8a65f4dc8ae88477": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_da403b31335640d1914c57c709ee0cbc",
       "max": 2,
       "style": "IPY_MODEL_856e0780a1bb45e0a9355947f6084824",
       "value": 2
      }
     },
     "d905464b4da24e35bfedae42c3eef534": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_b5ca196fe0a14af99966fc60ca78fc6b",
        "IPY_MODEL_c5f998bd592848d6a34210e43242c2d2"
       ],
       "layout": "IPY_MODEL_121741fb798246d4b526327ea488b022"
      }
     },
     "da03a67e2f7d4365a5d8460a1fe7bac3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "da403b31335640d1914c57c709ee0cbc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "ddc3b2718447459db97facd28200544f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_78e775f4f6ae45a08ae23f4d26221362",
       "max": 16,
       "style": "IPY_MODEL_c620f07561c2403ea6ab74be440e9697",
       "value": 16
      }
     },
     "e13fdd2c2801493fba598838e7346d69": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "VBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_31b2a6e102dc4140bd96f438334a0ea1",
        "IPY_MODEL_57fc909a20ab427f820a980e679bf6f0"
       ],
       "layout": "IPY_MODEL_42f1cac1905145b086b99aec1500ced9"
      }
     },
     "e655a826b7fb4c08bf5c8a6c7716488e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "eb194c6fd8534dbd808a17ab4a3d24a5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "ec5088958a4d464389421e0bc66f286b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "accuracy: 0.890625: 100%",
       "layout": "IPY_MODEL_c778782ad4de4ddb88c2f921b21f9d30",
       "max": 1,
       "style": "IPY_MODEL_8fd95eecc3034dc982a64c1a379405ca",
       "value": 1
      }
     },
     "ecf0026e5155445991c02373a5681b55": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "efe543f8e88f4022a2d232ba7a02d5d1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "f586be9e836348fb890cac4e388a9e17": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "fb306bd8776f459b8f32af1e73a0405a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "fd631ff022224ddba985e7fd3e3436d2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "VBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_d905464b4da24e35bfedae42c3eef534",
        "IPY_MODEL_4e4a7f9625a04f40802819b5464ffd62"
       ],
       "layout": "IPY_MODEL_2ae08a2cbeda4e20a191d96103824293"
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
